Quantization-aware training of quantized neural networks

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations for training a quantized neural network, the operations comprising, for each of a plurality of neural network layers and at each of a plurality of training time steps:

• receiving an input tensor for the neural network layer;
• processing the input tensor using the neural network layer to generate an output tensor, wherein the output tensor has a first precision;
• obtaining a current quantization range for output tensors of the neural network layer;
• processing the output tensor using the current quantization range to generate a quantized output tensor that has a second precision that is lower than the first precision;
• determining an error between the output tensor and the quantized output tensor;
• and
• determining an update to the quantization range using the determined error.

One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations for training a quantized neural network, the operations comprising, for each of a plurality of neural network layers and at each of a plurality of training time steps:

• receiving an input tensor for the neural network layer;
• processing the input tensor using the neural network layer to generate an output tensor, wherein the output tensor has a first precision;
• obtaining a current quantization range for output tensors of the neural network layer;
• processing the output tensor using the current quantization range to generate a quantized output tensor that has a second precision that is lower than the first precision;
• determining an error between the output tensor and the quantized output tensor;
• and
• determining an update to the quantization range using the determined error.