Quantizing Neural Networks Using Shifting and Scaling

A method for quantizing a neural network, the method comprising:

• receiving a definition of a neural network comprising a plurality of parameters at a plurality of layers, the parameters defined as floating point values;
• based on a distribution of input activation values for a particular layer of the neural network, selecting a set of scaling and shift values to be applied to input activation values of the layer in order for the input activation values to match a particular range of quantized values defined by a neural network inference circuit for which the neural network is trained;
• training the neural network using the selected set of scaling and shift values, wherein the parameters are defined as quantized values in the trained neural network;
• generating program instructions for the neural network inference circuit to execute the trained neural network with the quantized parameter values.

A non-transitory machine-readable medium storing a program which when executed by at least one processing unit quantizes a neural network, the program comprising sets of instructions for:

• receiving a definition of a neural network comprising a plurality of parameters at a plurality of layers, the parameters defined as floating point values;
• based on a distribution of input activation values for a particular layer of the neural network, selecting a set of scaling and shift values to be applied to input activation values of the layer in order for the input activation values to match a particular range of quantized values defined by a neural network inference circuit for which the neural network is trained;
• training the neural network using the selected set of scaling and shift values, wherein the parameters are defined as quantized values in the trained neural network;
• generating program instructions for the neural network inference circuit to execute the trained neural network with the quantized parameter values.