Training neural networks using linear companion nodes to mitigate gradient attenuation
AssigneeD5Ai LLC
InventorsJames K. Baker, Bradley J. Baker
Life of this patent
Estimating the term…
Abstract
A system and method for controlling a nodal network. The method includes estimating an effect on the objective caused by the existence or non-existence of a direct connection between a pair of nodes and changing a structure of the nodal network based at least in part on the estimate of the effect. A nodal network includes a strict partially ordered set, a weighted directed acyclic graph, an artificial neural network, and/or a layered feed-forward neural network.
Claims20
A computer-implemented method of training a neural network to reduce gradient attenuation during back-propagation, wherein the neural network comprises an input layer, an output layer, and at least one inner layer, wherein the at least one inner layer comprises at least one non-linear node, wherein the at least one non-linear node has a non-linear activation function, the method comprising:
- adding, by a computer system, a first linear node to the neural network, wherein the first linear node is initialized to receive the same inputs as the at least one non-linear node in the at least one inner layer of the neural network;
- adding, by the computer system, a second linear node to the neural network, wherein the second linear node receives a first input from the at least one non-linear node and a second input from the first linear node, wherein a connection weight from the first linear node to the second linear node is initialized to zero, and wherein each of the first and second linear nodes comprises a linear activation function;
- redirecting, by the computer system, outgoing connections of the at least one non-linear node to instead originate from the second linear node;
- and
- iteratively training, by the computer system, the neural network, wherein iteratively training the neural network comprises:
- back-propagating partial derivatives of an objective function through the neural network including through the first linear node, the at least one non-linear node, and the second linear node;
- and
- constraining a sum of summands to a constant value, wherein the summands comprise (i) a connection weight from the at least one non-linear node to the second linear node and (ii) the connection weight from the first linear node to the second linear node.
A computer system for training a neural network to reduce gradient attenuation during back-propagation, wherein the neural network comprises an input layer, an output layer, and at least one inner layer, wherein the at least one inner layer comprises at least one non-linear node, wherein the at least one non-linear node has a non-linear activation function, the computer system comprising:
- one or more processors;
- and
- computer memory in communication with the one or more processors, wherein the computer memory stores instructions that, when executed by the one or more processors, cause the one or more processors to:
- add a first linear node to the neural network, wherein the first linear node is initialized to receive the same inputs as the at least one non-linear node in the at least one inner layer of the neural network;
- add a second linear node to the neural network, wherein the second linear node receives a first input from the at least one non-linear node and a second input from the first linear node, wherein a connection weight from the first linear node to the second linear node is initialized to zero, and wherein each of the first and second linear nodes comprises a linear activation function;
- redirect outgoing connections of the at least one non-linear node to instead originate from the second linear node;
- and
- iteratively train the neural network by, among other steps:
- back-propagating partial derivatives of an objective function through the neural network including through the first linear node, the at least one non-linear node, and the second linear node;
- and
- constraining a sum of summands to a constant value, wherein the summands comprise (i) a connection weight from the at least one non-linear node to the second linear node and (ii) the connection weight from the first linear node to the second linear node.
Dependencies are inferred from the claim text.