# US 2025/0156716 A1 · Training neural networks using layerwise fisher approximations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0156716 A1
- Document ID: US20250156716A1
- Status: Published application
- Kind: A1
- Application number: 18/837,491
- Filed: 2023-02-10
- Published: 2025-05-15
- Earliest claimed: 2022-02-10
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ehsan Amid; Rohan Anil
- Web page: https://patentgrep.com/patent/US20250156716A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06F 7/78: for changing the order of data flow, e.g. matrix transposition or LIFO buffers; Overflow or underflow handling therefor
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 3/098: Distributed learning, e.g. federated learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network using layer-wise Fisher approximations.

## Claims

### Claim 1

A method for training a neural network having a plurality of neural network layers each having a respective set of weights to perform a machine learning task, the method comprising repeatedly performing training operations comprising:
obtaining a batch comprising one or more training inputs and a respective label for each training input;
performing a forward pass through the neural network and a backward pass through the neural network to determine:
(i) a respective layer input to each neural network layer for each training input,
(ii) a respective layer output for each neural network layer for each training input, and
(iii) for each neural network layer, a respective gradient with respect to the set of weights of the neural network layer of a task loss function for the machine learning task that includes one or more terms that measure, for each training input, a training output for the training input generated by performing the forward pass through the neural network relative to the respective label for the training input, and
for each neural network layer, updating the set of weights for the neural network layer by performing updating operations comprising:
determining, from the layer inputs to the neural network layer, a first estimate of a first expected self outer product of an input sampled from a distribution of layer inputs to the neural network layer;
determining, from the layer outputs for the neural network layer, a second estimate of a second expected self outer product of a gradient of a local loss for the layer for a layer output sampled from a distribution of layer outputs of the neural network layer;
determining an update to the set of weights for the neural network layer from the first estimate, the second estimate, and the respective gradient with respect to the task loss; and
updating the set of weights by applying the update to the set of weights.

### Claim 2 (depends on 1)

The method of claim 1, wherein the updating operations are performed in parallel for each of the plurality of neural network layers.

### Claim 3 (depends on 2)

The method of claim 2, wherein the updating operations for each of the neural network layers are assigned to and performed on a respective hardware device.

### Claim 4 (depends on 1)

The method of claim 1, wherein determining an update to the set of weights for the neural network layer from the first estimate, the second estimate, and the respective gradient with respect to the task loss comprises:
updating a first moving average using the first estimate;
updating a second moving average using the second estimate; and
determining the update to the set of weights for the neural network layer from the updated first moving average, the updated second moving average, and the respective gradient with respect to the task loss.

### Claim 5 (depends on 4)

The method of claim 4, wherein determining the update to the set of weights comprises:
determining a product of (i) an inverse of a matrix representation of the updated second moving average, (ii) a matrix representation of the respective gradient, and (iii) an inverse of a matrix representation of the updated first moving average.

### Claim 6 (depends on 5)

The method of claim 5, wherein determining the update further comprises multiplying the product by a step size value.

### Claim 7 (depends on 1)

The method of claim 1, wherein applying the update comprises subtracting the update from the weights.

### Claim 8 (depends on 1)

The method of claim 1, wherein determining, from the layer inputs to the neural network layer, a first estimate of a first expected self outer product of an input sampled from a distribution of layer inputs to the neural network layer comprises:
determining a combination of, for each layer input, a product between the layer input and a transpose of the layer input.

### Claim 9 (depends on 8)

The method of claim 8, wherein determining a combination of, for each layer input, a product between the layer input and a transpose of the layer input comprises:
determining a matrix product of (i) a transpose of a matrix having the layer inputs as the rows of the matrix and (ii) the matrix having the layer inputs as the rows of the matrix.

### Claim 10 (depends on 9)

The method of claim 9, wherein determining a combination of, for each layer input, a product between the layer input and a transpose of the layer input further comprises:
dividing each entry of the matrix product by a total number of training inputs in the batch.

### Claim 11 (depends on 1)

The method of claim 1, wherein determining, from the layer outputs for the neural network layer, a second estimate of a second expected self outer product of a gradient of a local loss for the layer for a layer output sampled from a distribution of layer outputs of the neural network layer comprises:
for each layer output, sampling a respective layer output from a corresponding probability distribution for the neural network layer;
for each layer output, computing a difference between the layer output and the respective sampled layer output; and
determining a combination of, for each layer output, a product between the difference for the layer output and a transpose of the difference for the layer output.

### Claim 12 (depends on 11)

The method of claim 11, wherein sampling from a corresponding probability distribution approximates sampling from a density induced by the transfer function for the layer.

### Claim 13 (depends on 11)

The method of claim 11, wherein for each layer output, sampling a respective layer output from a corresponding probability distribution for the neural network layer comprises sampling with a mean of the corresponding probability distribution set to be equal to the layer output.

### Claim 14 (depends on 11)

The method of claim 11, wherein determining a combination of, for each layer output, a product between the difference for the layer output and a transpose of the difference for the layer output comprises:
determining a matrix product of (i) a transpose of a matrix having the differences for the layer outputs as the rows of the matrix and (ii) the matrix having the differences for the layer outputs as the rows of the matrix.

### Claim 15 (depends on 14)

The method of claim 14, wherein determining a combination of, for each layer output, a product between the difference for the layer output and a transpose of the difference for the layer output further comprises:
dividing each entry of the matrix product by a total number of training inputs in the batch.

### Claim 16 (depends on 4)

The method of claim 4, wherein updating a first moving average using the first estimate comprises computing a weighted sum between the first moving average and the first estimate, and wherein updating a second moving average using the second estimate comprises computing a weighted sum between the second moving average and the second estimate.

### Claim 17

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations for training a neural network having a plurality of neural network layers each having a respective set of weights to perform a machine learning task, the method comprising repeatedly performing training operations comprising:
obtaining a batch comprising one or more training inputs and a respective label for each training input;
performing a forward pass through the neural network and a backward pass through the neural network to determine:
(i) a respective layer input to each neural network layer for each training input,
(ii) a respective layer output for each neural network layer for each training input, and
(iii) for each neural network layer, a respective gradient with respect to the set of weights of the neural network layer of a task loss function for the machine learning task that includes one or more terms that measure, for each training input, a training output for the training input generated by performing the forward pass through the neural network relative to the respective label for the training input, and
for each neural network layer, updating the set of weights for the neural network layer by performing updating operations comprising:
determining, from the layer inputs to the neural network layer, a first estimate of a first expected self outer product of an input sampled from a distribution of layer inputs to the neural network layer;
determining, from the layer outputs for the neural network layer, a second estimate of a second expected self outer product of a gradient of a local loss for the layer for a layer output sampled from a distribution of layer outputs of the neural network layer;
determining an update to the set of weights for the neural network layer from the first estimate, the second estimate, and the respective gradient with respect to the task loss; and
updating the set of weights by applying the update to the set of weights.

### Claim 18

One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations for training a neural network having a plurality of neural network layers each having a respective set of weights to perform a machine learning task, the method comprising repeatedly performing training operations comprising:
obtaining a batch comprising one or more training inputs and a respective label for each training input;
performing a forward pass through the neural network and a backward pass through the neural network to determine:
(i) a respective layer input to each neural network layer for each training input,
(ii) a respective layer output for each neural network layer for each training input, and
(iii) for each neural network layer, a respective gradient with respect to the set of weights of the neural network layer of a task loss function for the machine learning task that includes one or more terms that measure, for each training input, a training output for the training input generated by performing the forward pass through the neural network relative to the respective label for the training input, and
for each neural network layer, updating the set of weights for the neural network layer by performing updating operations comprising:
determining, from the layer inputs to the neural network layer, a first estimate of a first expected self outer product of an input sampled from a distribution of layer inputs to the neural network layer;
determining, from the layer outputs for the neural network layer, a second estimate of a second expected self outer product of a gradient of a local loss for the layer for a layer output sampled from a distribution of layer outputs of the neural network layer;
determining an update to the set of weights for the neural network layer from the first estimate, the second estimate, and the respective gradient with respect to the task loss; and
updating the set of weights by applying the update to the set of weights.

### Claim 19 (depends on 17)

The system of claim 17, wherein the updating operations are performed in parallel for each of the plurality of neural network layers.

### Claim 20 (depends on 19)

The system of claim 19, wherein the updating operations for each of the neural network layers are assigned to and performed on a respective hardware device.
