Methods and apparatuses for high performance and accuracy fixed-point batchnorm implementation

An apparatus for implementing a neural network, comprising:

• one or more processors of a hardware chip based on the neural network;
• and
• a memory configured to store instructions executable by the one or more processors;
• wherein the one or more processors, upon execution of the instructions, are configured to:
• receive floating-point input data over a channel of a standalone floating-point batchnorm layer in the neural network, and convert the floating-point input data into fixed-point input data of the standalone floating-point batchnorm layer;
• obtain fixed-point quantization parameters in each channel based on input data and three floating-point parameters μi, σi, εi in each channel;
• convert the standalone floating-point batchnorm layer based on the fixed-point quantization parameters into a fixed-point batchnorm layer in the neural network for processing the fixed-point input data to generate fixed-point output data;
• and
• map the fixed-point batchnorm layer to a fixed-point convolution layer, wherein computation of convolution is done by matrix multiplication executed on a GEMM engine, which is a hardwired unit in the hardware chip;
• wherein processing the fixed-point input data to generate the fixed-point output data comprises:
• multiplying the fixed-point input data with a first fixed-point quantization parameter to receive a first output;
• summing up the first output and a second fixed-point quantization parameter to receive a second output;
• right shifting with rounding the second output with an accumulator shift to receive a third output;
• clamping the third output to receive a fourth output;
• multiplying the fourth output with an output scale to receive a fifth output;
• right shifting with rounding the fifth output with an output shift to receive a sixth output;
• and
• clamping the sixth output into the fixed-point output data.

A non-transitory computer readable storage medium, comprising instructions stored therein to implement a neural network, wherein, upon execution of the instructions by one or more processors of a hardware chip based on the neural network, the instructions cause the one or more processors to perform acts comprising:

• receiving floating-point input data over a channel of a standalone floating-point batchnorm layer in the neural network, and converting the floating-point input data into fixed-point input data of the standalone floating-point batchnorm layer;
• obtaining fixed-point quantization parameters in each channel based on input data and three floating-point parameters μi, σi, εi in each channel;
• converting the standalone floating-point batchnorm layer based on the fixed-point quantization parameters into a fixed-point batchnorm layer in the neural network for processing the fixed-point input data to generate fixed-point output data;
• and
• mapping the fixed-point batchnorm layer to a fixed-point convolution layer, wherein computation of convolution is done by matrix multiplication that executed on a GEMM engine, which is a hardwired unit in the hardware chip;
• wherein processing the fixed-point input data to generate the fixed-point output data comprises:
• multiplying the fixed-point input data with a first fixed-point quantization parameter to receive a first output;
• summing up the first output and a second fixed-point quantization parameter to receive a second output;
• right shifting with rounding the second output with an accumulator shift to receive a third output;
• clamping the third output to receive a fourth output;
• multiplying the fourth output with an output scale to receive a fifth output;
• right shifting with rounding the fifth output with an output shift to receive a sixth output;
• and
• clamping the sixth output into the fixed-point output data.