Scaling half-precision floating point tensors for training deep neural networks

A non-transitory machine-readable medium to store instructions to cause one or more processors to perform operations comprising:

• executing, on a graphics processor, parallel threads of instructions associated with a command stream via a multiprocessor coupled with a memory controller, the multiprocessor having a single instruction, multiple thread (SIMT) architecture including hardware multi-threading, wherein the multiprocessor includes a mixed precision tensor processor;
• performing, via the mixed precision tensor processor, tensor computations associated with a layer of a neural network to generate error values, the error values stored in a floating point format;
• scaling the error values generated via the tensor computations by a scale factor to generate scaled error values;
• and
• performing a backward propagation operation associated with the layer of the neural network via the scaled error values to generate a scaled 16-bit floating point gradient tensor.