Scaling half-precision floating point tensors for training deep neural networks

A non-transitory machine-readable medium to store instructions to cause one or more processors to perform operations comprising:

• executing, on a graphics processor, parallel threads of instructions associated with a command stream via a multiprocessor coupled with a memory controller, the multiprocessor having a single instruction, multiple thread (SIMT) architecture including hardware multithreading, wherein the multiprocessor includes a set of functional units to execute at least one of the parallel threads of the instructions, the set of functional units including a mixed precision tensor processor;
• performing, via the mixed precision tensor processor, tensor computations associated with a layer of a neural network to generate loss data, the loss data stored in a first floating-point format;
• scaling the loss data generated via the tensor computations by a scaling factor to enable a data distribution of a gradient tensor generated based on the loss data to be represented by a scaled gradient tensor, the scaled gradient tensor stored as a second floating-point data type;
• and
• performing a backward propagation operation associated with the layer of the neural network to generate the scaled gradient tensor.