Back to results
US 11,436,496 B2Granted
Systems and methods for regularizing neural networks
AssigneeGoogle Inc.
InventorGil I. Shamir
Life of this patent
Estimating the term…
Abstract
The present disclosure relates generally to machine learning. More particularly, the present disclosure relates to systems and methods that regularize neural networks by decorrelating neurons or other parameters of the neural networks during training of the neural networks promoting these parameter to innovate over one another.
Claims21
A computer-implemented method to train neural networks, the method comprising:
- obtaining, by one or more computing devices, data descriptive of a neural network, the neural network comprising a plurality of layers of neurons;
- and
- backpropagating, by the one or more computing devices, a loss function through the neural network to train the neural network, wherein the loss function is descriptive of a performance of the neural network with respect to a set of training examples, and wherein backpropagating, by the one or more computing devices, the loss function through the neural network comprises, for each of one or more layers of the plurality of layers of the neural network:
- determining, by the one or more computing devices, a gradient of the loss function with respect to the neurons included in the layer, wherein, for at least the layer, the loss function includes an innovation loss term that provides, for each of one or more neurons included in the layer, a loss value that is based on an ability of one or more other neurons in the layer to predict a value of such neuron;
- and
- modifying, by the one or more computing devices, the neurons included in the layer based at least in part on the gradient of the loss function that includes the innovation loss term to decorrelate the neurons included in the layer.
3 more elements
A computer system to train neural networks, the computer system comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computer system to perform operations, the operations comprising:
- obtaining data descriptive of a neural network, the neural network comprising a plurality of layers of neurons;
- and
- determining for each of one or more neurons included in one or more of the plurality of layers, a benefit score based at least in part on an error of a prediction of a value of such neuron by one or more other neurons included in a same layer;
- and
- modifying a weight associated with at least one of the one or more neurons based at least in part on the benefit score determined for such at least one neuron.
5 more elements
One or more non-transitory computer-readable media that store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising:
- obtaining data descriptive of a neural network, the neural network comprising a plurality neurons respectively connected by a plurality of links;
- and
- backpropagating a loss function through the neural network to train the neural network, wherein the loss function is descriptive of a performance of the neural network with respect to a set of training examples, and wherein backpropagating the loss function through the neural network comprises, for each of one or more neurons, links, or biases of the neural network:
- determining a gradient of the loss function with respect to the one or more neurons, links, or biases of the neural network, wherein, for at least the one or more neurons, links, or biases of the neural network, the loss function includes an innovation loss term that provides, for each of the one or more neurons, links, or biases, a loss value that is based on an ability of one or more other neurons, links, or biases to predict a value of such neuron, link, or bias;
- and
- modifying the one or more neurons, links, or biases of the neural network based at least in part on the gradient of the loss function that includes the innovation loss term to decorrelate the one or more neurons, links, or biases of the neural network.
3 more elements
Dependencies are inferred from the claim text.