Accelerated training of a neural network via regularization
AssigneeD5Ai LLC
InventorsJames K. Baker, Bradley J. Baker
Life of this patent
Estimating the term…
Abstract
A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.
Claims33
A method of accelerating training a first neural network, wherein the first neural network comprises multiple nodes, wherein the multiple nodes comprise a first node, the method comprising:
- iterative training, by a programmed computer system that comprises one or more processor cores, via machine learning, the first neural network over a first number of iterations, wherein the iterative training comprises:
- for each training data item in a set of training data items:
- in a forward propagation phase, determining an activation values for each of the multiple nodes;
- and
- in a back-propagation phase, determining estimated partial derivatives of learned parameters and activation values for the multiple nodes with respect to, at least, an objective of the first neural network;
- and
- updating the learned parameters associated with the multiple nodes based on the estimated partial derivatives,
- wherein determining the estimated partial derivatives comprises determining estimated partial derivatives for the first node, and wherein determining the estimated partial derivatives for the first node comprises:
- adding by the programmed computer system, a regularization cost term to the objective for a node state of the first node for a first training data item in the set of training data items, wherein the regularization cost term is based on a difference between the node state of the first node for the first training data item and a node state of a second node for the first training data item, wherein the second node is one of multiple nodes of a second neural network that is separate from the first neural network.
A computer system for accelerated training of a first neural network, wherein the first neural network comprises multiple nodes, wherein the multiple nodes comprise a first node, the computer system comprising:
- one or more processor cores;
- and
- computer memory in communication with the one or more processor cores, wherein the computer stores instructions that when executed by the one or more processor cores cause the one or more processor cores to perform iterative training, via machine learning, of the first neural network, wherein the iterative training comprises:
- for each training data item in a set of training data items:
- in a forward propagation phase, determining an activation values for each of the multiple nodes;
- and
- in a back-propagation phase, determining estimated partial derivatives of learned parameters and activation values for the multiple nodes, including the first node, with respect to, at least, an objective of the first neural network;
- and
- updating the learned parameters associated with the multiple nodes based on the estimated partial derivatives,
- wherein the computer memory stores instructions that when executed by the one or more processor cores cause the one or more processor cores to determine the estimated partial derivatives for the first node by adding a regularization cost term to the objective for a node state of the first node for a first training data item in the set of training data items, wherein the regularization cost term is based on a difference between the node state of the first node for the first training data item and a node state of a second node for the first training data item, wherein the second node is one of multiple nodes of a second neural network that is separate from the first neural network.
Dependencies are inferred from the claim text.