Loss-scaling for deep neural network training with reduced precision

A computing arrangement comprising:

• a computer configured to perform computations associated with processing data through a deep neural network to develop a loss value and further configured to perform computations associated with back propagating the loss value through the deep neural network to compute gradients for weight updates,
• the computer being further configured to scale the loss value and to modify the computed gradients and/or at least one process that operates on said computed gradients to take the scale factor into account.

A method for training a deep neural network comprising:

• inputting a first hyperparameter specifying a scale factor for scaling computed gradients;
• and
• inputting a second hyperparameter that compensates a weight update for said scaling.

A method of modifying a deep neural network training system to permit lower precision computation of gradients while avoiding numerical computation problems associated with zeros and denormals due to use of said lower precision gradient computation, comprising:

• inserting a first code instruction(s) that scales at least one value used in a back propagation computation of said gradients;
• and
• inserting a second code instruction(s) that compensates a gradient-based weight update for said scaling.

A method of controlling a deep neural network training system to permit lower precision computation of gradients while avoiding numerical computation problems associated with zeros and denormals due to use of said lower precision gradient computation, comprising:

• inputting a first hyperparameter that scales at least one value used in a back propagation computation of said gradients;
• and
• inputting a second hyperparameter that compensates a gradient-based weight update for said scaling.

A deep neural network comprising layers each comprising at least one artificial neuron, each layer having a weight associated therewith,

• the weight having been trained by performing computations associated with processing of training data through the deep neural network to develop a loss value and back propagating the loss value through the deep neural network to compute, at reduced precision, a gradient used to update the weight, the contribution of the computed gradient to the trained weight having been adjusted to compensate for a scale factor used to enable computation of the gradient at said reduced precision while normalizing denormals and recovering zeros that would otherwise have occurred due to the reduced precision.