US 10,872,295 B1Granted

Residual quantization of bit-shift weights in an artificial neural network

AssigneeHong Kong Applied Science and Technology Institute Company Limited

InventorsYu Liu, Xuejiao Liu, Luhong Liang

Life of this patent

Estimating the term…

Loading patent…

Abstract

A neural network accelerator reads encoded weights from memory. All 1 bits in a weight except the first three are discarded. The first three leading 1 bits in the weight are encoded as three bit-shift values to form the encoded weight. The three bit-shift values are applied to a bit shifter to shift a node input to obtain three shifted inputs that are accumulated to generate the node output. Node complexity is reduced since only 3 shifts are performed rather than up to 15 shifts for a 16-bit weight. The bit shifter and accumulator for a node can be implemented by Look-Up Tables (LUTs) without requiring a Multiply-Accumulate (MAC) cell in a Field-Programmable Gate Array (FPGA). Quantization bias is reduced using a histogram analyzer that determines a weighted average for each interval between quantized weights. The third bit-shift value is incremented for weights in the interval above the weighted average.

Claims20

Dependent claims

A network system comprising:

a residual quantizer that receives a weight and encodes a bit-position of a leading 1 bit in the weight as a first bit-shift value, and encodes a bit-position of a second leading 1 bit in the weight as a second bit-shift value,
a bit-shift encoder that forms an encoded weight from the first bit-shift value and from the second bit-shift value, the bit-shift encoder discarding Least-Significant-Bits (LSBs) in the weight when forming the encoded weight;
a memory for storing the encoded weight, the memory storing a plurality of encoded weights for a plurality of nodes in the neural network;
a bit shifter that receives an input to a node in a neural network and generates a shifted input in response to a shift value;
an accumulator for accumulating the shifted input from the bit shifter to generate an accumulated result;
and
a bit-shift decoder that reads the encoded weight from the memory and applies the first bit-shift value extracted from the encoded weight to the bit shifter to accumulate a first shifted input into the accumulator, and that applies the second bit-shift value extracted from the encoded weight to the bit shifter to accumulate a second shifted input into the accumulator;
whereby weights are encoded as bit-shift values for leading 1 bits in the weight.

5 more elements

A neural network system comprising:

a weight quantizer that receives a weight for a node in a neural network and generates a first bit-shift value indicating a position of a leading 1 bit in the weight and generates a second bit-shift value indicating a position of a second leading 1 bit in the weight;
a residual quantizer that generates a third bit-shift value indicating a position of a third leading 1 bit in the weight;
a histogram analyzer that receives a plurality of weights for an interval between an upper quantized weight and a lower quantized weight;
wherein the upper quantized weight and the lower quantized weight each have all 0 bits for all bit positions of less significance than the position of the third leading 1 bit in the weight;
the histogram analyzer generating a weighted average of the plurality of weights for the interval, and activating an upper directional bit for all weights in the interval that are greater than the weighted average;
the residual quantizer incrementing the third bit-shift value for weights having the upper directional bit set;
a bit-shift encoder that encodes the first bit-shift value, the second bit-shift value, and the third bit-shift value to form an encoded weight, the bit-shift encoder not including in the encoded weight all bit positions of less significance than the position of the third leading 1 bit in the weight;
a bit shifter that shifts an input to the node by the first bit-shift value to generate a first shifted input, and that shifts the input to the node by the second bit-shift value to generate a second shifted input, and that shifts the input to the node by the third bit-shift value to generate a third shifted input;
and
an adder that adds together the first shifted input, the second shifted input, and the third shifted input to generate a result;
wherein the result is an approximation of an exact result of multiplying the input to the node by the weight for the node.

8 more elements

Dependencies are inferred from the claim text.