# US 12,182,685 B2 · Training a neural network using stochastic whitening batch normalization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,685 B2
- Document ID: US12182685B2
- Status: Granted
- Kind: B2
- Application number: 17/151,117
- Filed: 2021-01-16
- Published: 2022-04-21
- Granted: 2024-12-31
- Earliest claimed: 2020-03-24
- Assignees: LG Electronics Inc.
- Applicants: LG Electronics Inc.
- Inventors: Shengdong Zhang; Homa Fashandi; Ehsan Nezhadarya; Jiayi Liu
- Web page: https://patentgrep.com/patent/US12182685B2

## CPC

- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A neural network system, comprising: instructions for implementing at least a SWBN layer in a neural network, and wherein the instructions perform operations comprising: during training of the neural network system on a plurality of batches of training data and for each of the plurality of batches: obtaining a respective first layer output for each of the plurality of training data; determining a plurality of normalization statistics for the batch from the first layer outputs; generating a respective normalized output for each training data in the batch; updating the whitening matrix by a covariance matrix; performing stochastic whitening on the normalized components of each first layer output; transforming the whitened data for each training data; generating a respective SWBN layer output for each of the training data from the transformed whitened data for each training data in the batch; and providing the SWBN layer output.

## Claims

### Claim 1

A neural network system implemented by one or more processors, the neural network system comprising:
instructions for implementing at least a stochastic whitening batch normalization (SWBN) layer between a first neural network layer and a second neural network layer in a neural network, wherein the first neural network layer generates first layer outputs comprising a plurality of components, and wherein the instructions cause the one or more processors to perform operations comprising:
during training of the neural network system on a plurality of batches of training data and for each of the plurality of batches:
obtaining a respective first layer output for each of the plurality of training data in a batch;
determining a plurality of normalization statistics for the batch from the first layer outputs;
generating a respective normalized output for each training datum in the batch by normalizing each component of each first layer output using the determined plurality of normalization statistics;
updating a respective whitening matrix of the SWBN layer based on a whitening loss function using a covariance matrix determined based at least in part on the normalized output;
performing stochastic whitening on the normalized components of each first layer output to whiten data for each training datum in the batch using the updated respective whitening matrix of the SWBN layer;
transforming the whitened data for each training datum in the batch by using a first set of task parameters for rescaling the whitened data and a second set of task parameters for shifting the whitened data;
generating a respective SWBN layer output for each of the plurality of training data from the transformed whitened data for each training datum in the batch; and
providing the SWBN layer output as an input to the second neural network layer,
wherein the first and second sets of task parameters are decoupled from the whitening matrix such that:
the first and second sets of task parameters are updated in a backpropagation phase of the neural network based on a task loss function, and
the respective whitening matrix is only updated in a forward propagation phase based on the whitening loss function separate from the task loss function, and
wherein the SWBN layer is one of a plurality of SWBN layers of the neural network, and each SWBN layer of the plurality of SWBN layers has its own respective whitening matrix for performing stochastic whitening.

### Claim 2 (depends on 1)

The neural network system of claim 1, wherein the plurality of components of the first layer outputs are indexed by dimension, and wherein the plurality of normalization statistics are determined by:
determining, for each of the dimensions, a mean of the components of the first layer outputs in the dimension; and
determining, for each of the dimensions, a standard deviation of the components of the first layer outputs in the dimension.

### Claim 3 (depends on 2)

The neural network system of claim 2, wherein each component of each layer output is normalized by using the determined mean and determined standard deviation for the dimension corresponding to the component.

### Claim 4 (depends on 1)

The neural network system of claim 1, wherein the whitening loss function corresponds to a measure of distance between the covariance matrix and an identity matrix.

### Claim 5 (depends on 4)

The neural network system of claim 4, wherein the whitening loss function is derived by at least a Kullback-Leibler divergence or by a Frobenius norm of a difference between an identity matrix and a sample covariance matrix.

### Claim 6 (depends on 4)

The neural network system of claim 4, wherein the data for each training datum in the batch is whitened by multiplying the updated respective whitening matrix to the respective normalized output.

### Claim 7 (depends on 1)

The neural network system of claim 1, wherein rescaling the whitened data further comprises multiplying each of the parameters in the first set of task parameters to each component of the whitened data that corresponds to each dimension, and wherein shifting the whitened data further comprises adding each parameter in the second set of task parameters to each component of the whitened data that corresponds to each dimension.

### Claim 8 (depends on 1)

The neural network system of claim 1, wherein the first set of task parameters and the second set of task parameters remain static in the forward propagation phase.

### Claim 9 (depends on 1)

The neural network system of claim 1, wherein, during the training of the neural network, the neural network system is configured to store the normalization statistics as part of adjusting gradients, wherein the gradients are adjusted to update the first and second sets of task parameters for rescaling and shifting the whitened data in the SWBN layer.

### Claim 10 (depends on 9)

The neural network system of claim 9, wherein updating normalization statistics has no effect on other parameters in the neural network system.

### Claim 11

A method performed by one or more computers implementing a stochastic whitening batch normalization (SWBN) layer that is between a first neural network layer and a second neural network layer in a neural network system, wherein the first neural network generates first layer outputs having a plurality of components, and wherein the method comprises:
during training of the neural network system on a plurality of batches of training data and for each of the plurality of batches:
obtaining a respective first layer output for each of the plurality of training data in a batch;
determining a plurality of normalization statistics for the batch from the first layer outputs;
generating a respective normalized output for each training datum in the batch by normalizing each component of each first layer output using the determined plurality of normalization statistics;
updating a respective whitening matrix of the SWBN layer based on a whitening loss function using a covariance matrix determined based at least in part on the normalized output;
performing stochastic whitening on the normalized components of each first layer output to whiten data for each training datum in the batch using the updated respective whitening matrix of the SWBN layer;
transforming the whitened data for each training datum in the batch by using a first set of task parameters for rescaling the whitened data and a second set of task parameters for shifting the whitened data;
generating a respective SWBN layer output for each of the plurality of training data from the transformed whitened data for each training datum in the batch; and
providing the SWBN layer output as an input to the second neural network layer,
wherein the first and second sets of task parameters are decoupled from the whitening matrix such that:
the first and second sets of task parameters are updated in a backpropagation phase of the neural network based on a task loss function, and
the respective whitening matrix is only updated in a forward propagation phase based on the whitening loss function separate from the task loss function, and
wherein the SWBN layer is one of a plurality of SWBN layers of the neural network, and each SWBN layer of the plurality of SWBN layers has its own respective whitening matrix for performing stochastic whitening.

### Claim 12 (depends on 11)

The method of claim 11, wherein the plurality of components of the first layer outputs are indexed by dimension, and wherein the plurality of normalization statistics are determined by:
determining, for each of the dimensions, a mean of the components of the first layer outputs in the dimension; and
determining, for each of the dimensions, a standard deviation of the components of the first layer outputs in the dimension.

### Claim 13 (depends on 12)

The method of claim 12, wherein each component of each layer output is normalized by using the determined mean and determined standard deviation for the dimension corresponding to the component.

### Claim 14 (depends on 12)

The method of claim 12, wherein the whitening loss function corresponds to a measure of distance between the covariance matrix and an identity matrix.

### Claim 15 (depends on 14)

The method of claim 14, wherein the data for each training datum in the batch is whitened by multiplying the updated respective whitening matrix to the respective normalized output.

### Claim 16

A non-transitory memory storing one or more programs, which, when executed by one or more processors of a device, cause the device to be configured to perform operations to implement a neural network, wherein the neural network comprises:
a stochastic whitening batch normalization (SWBN) layer between a first neural network layer and a second neural network layer, wherein the first neural network layer generates first layer outputs having a plurality of components, and wherein the SWBN layer is configured to:
during training of a neural network system on a plurality of batches of training data and for each of the plurality of batches:
obtaining a respective first layer output for each of the plurality of training data in a batch;
determining a plurality of normalization statistics for the batch from the first layer outputs;
generating a respective normalized output for each training datum in the batch by normalizing each component of each first layer output using the determined plurality of normalization statistics;
updating a respective whitening matrix of the SWBN layer based on a whitening loss function using a covariance matrix determined based at least in part on the normalized output;
performing stochastic whitening on the normalized components of each first layer output to whiten data for each training datum in the batch using the updated respective whitening matrix of the SWBN layer;
transforming the whitened data for each training datum in the batch by using a first set of task parameters for rescaling the whitened data and a second set of task parameters for shifting the whitened data;
generating a respective SWBN layer output for each of the plurality of training data from the transformed whitened data for each training datum in the batch; and
providing the SWBN layer output as an input to the second neural network layer,
wherein the first and second sets of task parameters are decoupled from the whitening matrix such that:
the first and second sets of task parameters are updated in a backpropagation phase of the neural network based on a task loss function, and
the respective whitening matrix is only updated in a forward propagation phase based on the whitening loss function separate from the task loss function, and
wherein the SWBN layer is one of a plurality of SWBN layers of the neural network, and each SWBN layer of the plurality of SWBN layers has its own respective whitening matrix for performing stochastic whitening.