Scaling half-precision floating point tensors for training deep neural networks

A method comprising:

• concurrently executing multiple thread groups via a streaming multiprocessor of a graphics processor, the streaming multiprocessor including a single instruction, multiple thread (SIMT) architecture;
• performing tensor operations via first circuitry of the streaming multiprocessor, the tensor operations associated with a first layer of a neural network;
• and
• via second circuitry of the streaming multiprocessor:
• analyzing statistics for output values of the tensor operations;
• determining a target format to convert the output values, the target format determined based on the statistics for the output values and a precision associated with a second layer of the neural network;
• and
• converting the output values to the target format.

A data processing system comprising:

• a memory device;
• and
• a graphics processor coupled with the memory device, the graphics processor comprising:
• a level-two (L2) cache configured to cache data stored to the memory device;
• a memory controller coupled with the memory device and the L2 cache;
• and
• a streaming multiprocessor coupled with the memory controller, the streaming multiprocessor to concurrently execute multiple thread groups, wherein the streaming multiprocessor includes a single instruction, multiple thread (SIMT) architecture, the streaming multiprocessor including:
• first circuitry to perform tensor operations associated with a first layer of a neural network;
• and
• second circuitry to:
• analyze statistics for output values of the tensor operations;
• determine a target format to convert the output values, the target format determined based on the statistics for the output values and a precision associated with a second layer of the neural network;
• and
• convert the output values to the target format.