Adjusting activation compression for neural network training
AssigneeMicrosoft Technology Licensing, LLC
InventorsDaniel Lo, Bita Darvish Rouhani, Eric S. Chung, Yiren Zhao, Amar Phanishayee, Ritchie Zhao
Life of this patent
Estimating the term…
Loading patent…
AssigneeMicrosoft Technology Licensing, LLC
InventorsDaniel Lo, Bita Darvish Rouhani, Eric S. Chung, Yiren Zhao, Amar Phanishayee, Ritchie Zhao
Estimating the term…
Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and, in particular, for adjusting floating-point formats used to store activation values during training. In certain examples of the disclosed technology, a computing system includes processors, memory, and a floating-point compressor in communication with the memory. The computing system is configured to produce a neural network comprising activation values expressed in a first floating-point format, select a second floating-point format for the neural network based on a performance metric, convert at least one of the activation values to the second floating-point format, and store the compressed activation values in the memory. Aspects of the second floating-point format that can be adjusted include the number of bits used to express mantissas, exponent format, use of non-uniform mantissas, and/or use of outlier values to express some of the mantissas.
A computing system comprising:
A quantization-enabled system comprising:
A method, implemented in a computing system comprising one or more processors, bulk memory comprising computer-readable storage devices and/or memory, and a floating-point compressor formed from at least one of the processors, the floating-point compressor being in communication with the bulk memory, the method comprising:
Dependencies are inferred from the claim text.