# US 2018/0322391 A1 · Loss-scaling for deep neural network training with reduced precision

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322391 A1
- Document ID: US20180322391A1
- Status: Published application
- Kind: A1
- Application number: 15/971,884
- Filed: 2018-05-04
- Published: 2018-11-08
- Earliest claimed: 2017-05-05
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Hao WU; Jonah ALBEN; Paulius MICIKEVICIUS
- Web page: https://patentgrep.com/patent/US20180322391A1

## CPC

- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0499: Feedforward networks
- G06N 3/063: using electronic means
- G06N 3/09: Supervised learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In training a deep neural network using reduced precision, gradient computation operates on larger values without affecting the rest of the training procedure. One technique trains the deep neural network to develop loss, scales the loss, computes gradients at a reduced precision, and reduces the magnitude of the computed gradients to compensate for scaling of the loss. In one example non-limiting arrangement, the training forward pass scales a loss value by some factor S and the weight update reduces the weight gradient contribution by 1/S. Several techniques can be used for selecting scaling factor S and adjusting the weight update.

## Claims

### Claim 1

A system for training a neural network comprising:
at least one numerical computation circuit configurable to compute gradients based on a loss value; and
a reduced precision selector coupled to said at least one numerical computation circuit, said reduced precision selector controlling said at least one numerical computation circuit to selectively operate in reduced precision computation mode;
wherein the at least one numerical computation circuit is further configured to scale said loss value when adjusting weights based on gradients computed using said reduced precision mode.

### Claim 2 (depends on 1)

The system of claim 1 wherein the at least one numerical computation circuit is configured to compensate for said scaling by reducing the weight gradient contribution in a way that is inversely proportional to said scaling.

### Claim 3 (depends on 1)

The system of claim 1 wherein said at least one numerical computation circuit is configured to scale said loss value based at least in part on a hyperparameter.

### Claim 4 (depends on 1)

The system of claim 1 wherein said at least one numerical computation circuit is configured to scale said loss value by an automatically-selected scaling factor.

### Claim 5 (depends on 1)

The system of claim 1 further including at least one processor configured to automatically select the scaling factor for each iteration of training of said neural network based on a largest magnitude weight gradient determined in a last iteration.

### Claim 6 (depends on 5)

The system of claim 5 wherein said at least one processor computes an upper bound on the scaling factor u=log2(215−x).

### Claim 7 (depends on 6)

The system of claim 6 wherein said at least one processor further reduces the scaling factor by a constant to prevent overflow.

### Claim 8 (depends on 6)

The system of claim 6 wherein the at least one processor is further configured to test weight gradients determined by a training iteration of the neural network, and repeating the iteration with a reduced scaling factor conditioned on the results of the test.

### Claim 9 (depends on 1)

The system of claim 1 wherein the at least one numerical computation circuit modifies each weight gradient value by an amount inversely proportional to the scaling factor before using the weight gradient value for a weight update.

### Claim 10 (depends on 1)

The system of claim 1 wherein the at least one numerical computation circuit performs weight updates by combining the gradients with a further parameter that is adjusted based on the scaling.

### Claim 11 (depends on 10)

The system of claim 10 wherein the parameter comprises learning rate.

### Claim 12 (depends on 10)

The system of claim 10 wherein the parameter comprises gradient clipping threshold.

### Claim 13 (depends on 10)

The system of claim 10 wherein the parameter comprises weight decay.

### Claim 14

A process of training a deep neural network comprising:
(a) forward processing data through the deep neural network to develop a loss value;
(b) scaling the loss value by a scale factor;
(c) back propagating the scaled loss value through the deep neural network to compute gradients; and
(d) adjusting weights of the deep neural network based on the computed gradients.

### Claim 15 (depends on 14)

The process of claim 14 wherein the adjusting includes reducing the computed gradients by the scale factor before using them to adjust the weights.

### Claim 16 (depends on 14)

The process of claim 14 further including computing the gradients using reduced precision.

### Claim 17 (depends on 14)

The process of claim 14 further including computing the gradients at a lower precision than is used for at least some computations associated with the forward processing of training data.

### Claim 18 (depends on 14)

The process of claim 14 further including computing gradients using half precision while recovering zeroes and normalizing denormals due to said half precision.

### Claim 19

A non-transitory memory storing instructions that when executed by at least one processor control the at least one processor to perform steps comprising:
(a) processing data with a deep neural network to develop a loss value;
(b) scaling the loss value by a scale factor;
(c) back propagating the scaled loss value through the deep neural network to compute gradients; and
(d) using the computed gradients to adjust weights of the deep neural network.

### Claim 20

A mixed precision computing component comprising:
at least one numerical computation circuit capable of operating in a reduced precision mode;
the at least one numerical computation circuit being configured to perform computations associated with processing data through a deep neural network to develop a loss value; and
when operating in the reduced precision mode, the at least one numerical computation circuit being configured to perform computations associated with back propagating the loss value through the deep neural network to compute gradients useful for weight updates, wherein the at least one numerical computation circuit is further configured to scale the loss value.

### Claim 21 (depends on 20)

The component of claim 20 wherein the numerical computation circuit is configured to develop the loss value using said reduced precision mode, and to compute weight updates using other than said reduced precision mode.

### Claim 22

A computing arrangement comprising:
a computer configured to perform computations associated with processing data through a deep neural network to develop a loss value and further configured to perform computations associated with back propagating the loss value through the deep neural network to compute gradients for weight updates,
the computer being further configured to scale the loss value and to modify the computed gradients and/or at least one process that operates on said computed gradients to take the scale factor into account.

### Claim 23

A method for training a deep neural network comprising:
inputting a first hyperparameter specifying a scale factor for scaling computed gradients; and
inputting a second hyperparameter that compensates a weight update for said scaling.

### Claim 24 (depends on 22)

The method of claim 22 wherein the compensating comprises multiplying or dividing the computed gradients by said second hyperparameter.

### Claim 25 (depends on 22)

The method of claim 22 wherein the compensating comprises modifying a learning rate in accordance with said second hyperparameter.

### Claim 26

A method of modifying a deep neural network training system to permit lower precision computation of gradients while avoiding numerical computation problems associated with zeros and denormals due to use of said lower precision gradient computation, comprising:
inserting a first code instruction(s) that scales at least one value used in a back propagation computation of said gradients; and
inserting a second code instruction(s) that compensates a gradient-based weight update for said scaling.

### Claim 27 (depends on 26)

The method of claim 26 wherein the at least one value comprises a loss value, and the first code instruction(s) automatically develop a scale factor used to scale the loss value.

### Claim 28

A method of controlling a deep neural network training system to permit lower precision computation of gradients while avoiding numerical computation problems associated with zeros and denormals due to use of said lower precision gradient computation, comprising:
inputting a first hyperparameter that scales at least one value used in a back propagation computation of said gradients; and
inputting a second hyperparameter that compensates a gradient-based weight update for said scaling.

### Claim 29

A deep neural network comprising layers each comprising at least one artificial neuron, each layer having a weight associated therewith,
the weight having been trained by performing computations associated with processing of training data through the deep neural network to develop a loss value and back propagating the loss value through the deep neural network to compute, at reduced precision, a gradient used to update the weight, the contribution of the computed gradient to the trained weight having been adjusted to compensate for a scale factor used to enable computation of the gradient at said reduced precision while normalizing denormals and recovering zeros that would otherwise have occurred due to the reduced precision.

### Claim 30

A method comprising iteratively:
training at a first precision to develop loss;
scaling the loss;
computing gradients at a second precision lower than the first precision; and
reducing the magnitude of the computed gradients to compensate for scaling of the loss.

### Claim 31

A method comprising iteratively:
forward propagating training data through a deep neural network to develop a loss value;
back propagating the loss value through the deep neural network to develop weight gradients; and
configuring the back propagating to recover zeros and normalize denormals without adversely affecting a subsequent weight update based on the weight gradients.

### Claim 32 (depends on 20)

The component of claim 20 wherein the scaling compensates for a magnitude component of the computed gradients due to said scaling.
