# US 12,008,467 B2 · Asymmetric quantization for compression and for acceleration of inference for neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,467 B2
- Document ID: US12008467B2
- Status: Granted
- Kind: B2
- Application number: 16/877,582
- Filed: 2020-05-19
- Published: 2021-01-07
- Granted: 2024-06-11
- Earliest claimed: 2019-07-01
- Assignees: Baidu USA LLC
- Applicants: Baidu USA, LLC
- Inventors: Yingzhen Yang; Zhibiao Zhao; Baoxin Zhao; Jun Huan; Jian Ouyang; Yong Wang; Jiaxin Shi
- Web page: https://patentgrep.com/patent/US12008467B2

## CPC

- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Presented herein are embodiments of an improved asymmetric quantization, which may generally be referred to as improved asymmetric quantization (IAQ) embodiments. IAQ embodiments combine the benefits of conventional asymmetric quantization and symmetric quantization but also provide additional computation efficiencies. Embodiments of IAQ adopt an asymmetric range of the weights of a neural network layer, so they circumvent the limitation of symmetric range of symmetric quantization. Moreover, the inference process of a neural network quantized by an IAQ embodiment is much faster than that of the neural network quantized by conventional asymmetric quantization by quantizing an offset value of each layer.

## Claims

### Claim 1

A computer-implemented method for asymmetric quantization of at least some weight values of a neural network, the method comprising:
identifying a set of extrema weight values from weight values for a layer of the neural network, the set of extrema weight values comprising a maximum weight value and a minimum weight value;
obtaining a scaling factor for quantizing the weight values of the layer of the neural network using the set of extrema weight values and a number of bits that will be used to represent the weight values in quantized form;
using one of the extrema weight values and the scaling factor to obtain an offset value for the layer by performing steps comprising:
dividing the extreme weight value by the scaling factor to obtain a quotient; and
converting the quotient to an integer value;
using the scaling factor and the extreme weight value from the set of extrema weight values that was used to obtain the offset value to quantize the weight values for the layer; and
for the layer, storing the scaling factor, the offset value, and the quantized weight values, to be used during inference, in which obtaining an output for the layer comprises using only integer operations to adjust the quantized weight values by the offset value and to multiply the adjusted quantized weight values with input values for the layer.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 further comprising:
computing an output for the layer of the neural network using the scaling factor for the layer, the offset value for the layer, the quantized weight values for the layer, and input values, wherein integer operations are used to adjust the quantized weight values by the offset value and to multiply the adjusted quantized weight values with the input values.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1 wherein the extreme weight value is the maximum weight value and an adjusted quantized weight value is obtained by subtracting a quantized weight value from the offset value using an integer operation.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1 wherein the extreme weight value is the minimum weight value and an adjusted quantized weight value is obtained by adding a quantized weight value to the offset value using an integer operation.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1 wherein the step of converting the quotient to an integer value comprises using rounding or truncation.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1 further comprising:
using the method of claim 1 for each layer of two or more layers of the neural network.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6 wherein at least a plurality of the two or more layers are consecutive layers and the method further comprises:
computing an output for the consecutive layers of the neural network by performing the steps comprising:
using integer operations to:
obtain, for each layer of the consecutive layers, a set of adjusted quantized weight values for the layer by adjusting the quantized weight values of the layer by the offset value of the layer; and
multiply together the sets of adjusted quantized weight values and input values for the first layer of the consecutive layers to obtain an intermediate product; and
multiplying together the scaling factors for each layer of the consecutive layers and the intermediate product to obtain the output for the consecutive layers.

### Claim 8

A system comprising:
one or more processors; and
a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps for asymmetric quantization of at least some weight values of a neural network to be performed comprising:
identifying a set of extrema weight values from weight values for a layer of the neural network, the set of extrema weight values comprising a maximum weight value and a minimum weight value;
obtaining a scaling factor for quantizing the weight values of the layer of the neural network using the set of extrema weight values and a number of bits that will be used to represent the weights values in quantized form;
using one of the extrema weight values and the scaling factor to obtain an offset value for the layer by performing steps comprising:
dividing the extreme weight value by the scaling factor to obtain a quotient;
and
converting the quotient to an integer value;
using the scaling factor and the extreme weight value from the set of extrema weight values that was used to obtain the offset value to quantize the weight values for the layer; and
for the layer, storing the scaling factor, the offset value, and the quantized weight values, to be used during inference, in which obtaining an output for the layer comprises using only integer operations to adjust the quantized weight values by the offset value and to multiply the adjusted quantized weight values with input values for the layer.

### Claim 9 (depends on 8)

The system of claim 8 wherein the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
computing an output for the layer of the neural network using the scaling factor for the layer, the offset value for the layer, the quantized weight values for the layer, and input values, wherein integer operations are used to adjust the quantized weight values by the offset value and to multiply the adjusted quantized weight values with the input values.

### Claim 10 (depends on 8)

The system of claim 8 wherein the extreme weight value is the maximum weight value and an adjusted quantized weight value is obtained by subtracting a quantized weight value from the offset value using an integer operation.

### Claim 11 (depends on 8)

The system of claim 8 wherein the extreme weight value is the minimum weight value and an adjusted quantized weight value is obtained by adding a quantized weight value to the offset value using an integer operation.

### Claim 12 (depends on 8)

The system of claim 8 wherein the step of converting the quotient to an integer value comprises using rounding or truncation.

### Claim 13 (depends on 8)

The system of claim 8 wherein the steps of claim 8 are applied for each layer of two or more layers of the neural network.

### Claim 14 (depends on 13)

The system of claim 13 wherein at least a plurality of the two or more layers are consecutive layers and the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
computing an output for the consecutive layers of the neural network by performing the steps comprising:
using integer operations to:
obtain, for each layer of the consecutive layers, a set of adjusted quantized weight values for the layer by adjusting the quantized weight values of the layer by the offset value of the layer; and
multiply together the sets of adjusted quantized weight values and input values for the first layer of the consecutive layers to obtain an intermediate product; and
multiplying together the scaling factors for each layer of the consecutive layers and the intermediate product to obtain the output for the consecutive layers.

### Claim 15

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by one or more processors, causes steps for asymmetric quantization of at least some weight values of a neural network to be performed comprising:
identifying a set of extrema weight values from weight values for a layer of the neural network, the set of extrema weight values comprising a maximum weight value and a minimum weight value;
obtaining a scaling factor for quantizing the weight values of the layer of the neural network using the set of extrema weight values and a number of bits that will be used to represent the weights values in quantized form;
using one of the extrema weight values and the scaling factor to quantize the weight values for the layer of the neural network;
obtaining an offset value for the layer that is an integer value using the scaling factor and the extreme weight value from the set of extrema weight values that was used to quantize the weight values for the layer by performing steps comprising:
dividing the extreme weight value by the scaling factor to obtain a quotient;
and
converting the quotient to the integer value; and
for the layer, storing the scaling factor, the offset value, and the quantized weight values, to be used during inference, in which obtaining an output for the layer comprises using only integer operations to adjust the quantized weight values by the offset value and to multiply the adjusted quantized weight values with input values for the layer.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 further comprising one or more sequences of instructions which, when executed by one or more processors, causes steps to be performed comprising:
computing an output for the layer of the neural network using the scaling factor for the layer, the offset value for the layer, the quantized weights for the layer, and input values, wherein integer operations are used to adjust the quantized weights by the offset value and to multiply the adjusted quantized weight values with the input values.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 wherein the extreme weight value is the maximum weight value and an adjusted quantized weight value is obtained by subtracting a quantized weight value from the offset value using an integer operation.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 wherein the extreme weight value is the minimum weight value and an adjusted quantized weight value is obtained by adding a quantized weight value to the offset value using an integer operation.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 wherein the step of converting the quotient to the integer value comprises using rounding or truncation.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium or media of claim 15 wherein the neural network comprise a set of consecutive layers and the non-transitory computer-readable medium or media further comprises one or more sets of instructions which, when executed by one or more processors, causes steps to be performed comprising:
computing an output for the set of consecutive layers of the neural network by performing the steps comprising:
using integer operations to:
obtain, for each layer of the set of consecutive layers, a set of adjusted quantized weight values for the layer by adjusting the quantized weight values of the layer by the offset value of the layer; and
multiply together the sets of adjusted quantized weight values and input values for the first layer of the set of consecutive layers to obtain an intermediate product; and
multiplying together the scaling factors for each layer of the set of consecutive layers and the intermediate product to obtain the output for the set of consecutive layers.