Training neural networks using layerwise fisher approximations
AssigneeGoogle Inc.
InventorsEhsan Amid, Rohan Anil
Life of this patent
Estimating the term…
Abstract
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network using layer-wise Fisher approximations.
Claims20
A method for training a neural network having a plurality of neural network layers each having a respective set of weights to perform a machine learning task, the method comprising repeatedly performing training operations comprising:
- obtaining a batch comprising one or more training inputs and a respective label for each training input;
- performing a forward pass through the neural network and a backward pass through the neural network to determine:
- (i) a respective layer input to each neural network layer for each training input,
- (ii) a respective layer output for each neural network layer for each training input, and
- (iii) for each neural network layer, a respective gradient with respect to the set of weights of the neural network layer of a task loss function for the machine learning task that includes one or more terms that measure, for each training input, a training output for the training input generated by performing the forward pass through the neural network relative to the respective label for the training input, and
- for each neural network layer, updating the set of weights for the neural network layer by performing updating operations comprising:
- determining, from the layer inputs to the neural network layer, a first estimate of a first expected self outer product of an input sampled from a distribution of layer inputs to the neural network layer;
- determining, from the layer outputs for the neural network layer, a second estimate of a second expected self outer product of a gradient of a local loss for the layer for a layer output sampled from a distribution of layer outputs of the neural network layer;
- determining an update to the set of weights for the neural network layer from the first estimate, the second estimate, and the respective gradient with respect to the task loss;
- and
- updating the set of weights by applying the update to the set of weights.
A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations for training a neural network having a plurality of neural network layers each having a respective set of weights to perform a machine learning task, the method comprising repeatedly performing training operations comprising:
- obtaining a batch comprising one or more training inputs and a respective label for each training input;
- performing a forward pass through the neural network and a backward pass through the neural network to determine:
- (i) a respective layer input to each neural network layer for each training input,
- (ii) a respective layer output for each neural network layer for each training input, and
- (iii) for each neural network layer, a respective gradient with respect to the set of weights of the neural network layer of a task loss function for the machine learning task that includes one or more terms that measure, for each training input, a training output for the training input generated by performing the forward pass through the neural network relative to the respective label for the training input, and
- for each neural network layer, updating the set of weights for the neural network layer by performing updating operations comprising:
- determining, from the layer inputs to the neural network layer, a first estimate of a first expected self outer product of an input sampled from a distribution of layer inputs to the neural network layer;
- determining, from the layer outputs for the neural network layer, a second estimate of a second expected self outer product of a gradient of a local loss for the layer for a layer output sampled from a distribution of layer outputs of the neural network layer;
- determining an update to the set of weights for the neural network layer from the first estimate, the second estimate, and the respective gradient with respect to the task loss;
- and
- updating the set of weights by applying the update to the set of weights.
One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations for training a neural network having a plurality of neural network layers each having a respective set of weights to perform a machine learning task, the method comprising repeatedly performing training operations comprising:
- obtaining a batch comprising one or more training inputs and a respective label for each training input;
- performing a forward pass through the neural network and a backward pass through the neural network to determine:
- (i) a respective layer input to each neural network layer for each training input,
- (ii) a respective layer output for each neural network layer for each training input, and
- (iii) for each neural network layer, a respective gradient with respect to the set of weights of the neural network layer of a task loss function for the machine learning task that includes one or more terms that measure, for each training input, a training output for the training input generated by performing the forward pass through the neural network relative to the respective label for the training input, and
- for each neural network layer, updating the set of weights for the neural network layer by performing updating operations comprising:
- determining, from the layer inputs to the neural network layer, a first estimate of a first expected self outer product of an input sampled from a distribution of layer inputs to the neural network layer;
- determining, from the layer outputs for the neural network layer, a second estimate of a second expected self outer product of a gradient of a local loss for the layer for a layer output sampled from a distribution of layer outputs of the neural network layer;
- determining an update to the set of weights for the neural network layer from the first estimate, the second estimate, and the respective gradient with respect to the task loss;
- and
- updating the set of weights by applying the update to the set of weights.
Dependencies are inferred from the claim text.