# US 2024/0004952 A1 · Hardware-Aware Mixed-Precision Quantization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004952 A1
- Document ID: US20240004952A1
- Status: Published application
- Kind: A1
- Application number: 17/852,484
- Filed: 2022-06-29
- Published: 2024-01-04
- Earliest claimed: 2022-06-29
- Assignees: MediaTek Singapore Pte. Ltd.
- Applicants: MediaTek Singapore Pte. Ltd.
- Inventors: Hantao Huang; Ziang Yang; Jia Yao Christopher Lim; Jung Hau Foo; Chia-Lin Yu
- Web page: https://patentgrep.com/patent/US20240004952A1

## CPC

- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06N 3/02: Neural networks
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A bit-widths determination method selects bit-widths for mixed-precision neural network computing on a target hardware platform. An activation quantization sensitivity (AQS) value is calculated for each convolution layer in a neural network. The AQS value indicates the sensitivity of convolution output to quantized convolution input. One or more convolution layers are grouped into a quantization group, which is to be executed by a corresponding set of target hardware. A group AQS value is calculated for each quantization group based on the AQS values of the convolution layers in the quantization group. Then bit-widths supported by the target hardware platform are selected for the corresponding quantization groups. The bit-widths are selected to optimize, under a given constraint, a sensitivity metric that is calculated based on each quantization group's group AQS value.

## Claims

### Claim 1

A method for determining bit-widths for mixed-precision neural network computing on a target hardware platform, comprising:
calculating an activation quantization sensitivity (AQS) value for each of a plurality of convolution layers in a neural network, wherein the AQS value indicates sensitivity of convolution output to quantized convolution input;
forming a plurality of quantization groups by grouping one or more convolution layers into a quantization group to be executed by a corresponding set of target hardware;
calculating a group AQS value for each quantization group based on AQS values of the convolution layers in the quantization group;
selecting bit-widths supported by the target hardware platform for corresponding quantization groups to optimize, under a given constraint, a sensitivity metric that is calculated based on each quantization group's group AQS value.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
performing Integer Linear Programming (ILP) to obtain a mixed-precision quantization configuration for the neural network that minimizes a total AQS under the given constraint, wherein the total AQS is a sum of all group AQS values in the neural network.

### Claim 3 (depends on 2)

The method of claim 2, wherein the given constraint is one of a model size, latency, and total binary operations of the neural network.

### Claim 4 (depends on 1)

The method of claim 1, wherein the given constraint is total binary operations (BOPS), which limits a sum of per-group BOPS, and wherein per-group BOPS is calculated as the number of MAC operations multiplied by the bit-width of input activation and the bit-width of weights.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
selecting an input activation bit-width and a weight bit-width for each quantization group to optimize the sensitivity metric under the given constraint.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
identifying one or more quantization groups to lower corresponding bit-widths under a user-defined accuracy criterion.

### Claim 7 (depends on 6)

The method of claim 6, wherein an identified quantization group is one having a highest group quality
log
⁢
(
MAC
+
1
)
goup
⁢
AQS
among all quantization groups, wherein MAC represents a number of multiply-and-add operations.

### Claim 8 (depends on 1)

The method of claim 1, wherein the group AQS value is the maximum AQS value of all convolution layers in the quantization group.

### Claim 9 (depends on 1)

The method of claim 1, wherein calculating the AQS value further comprises:
calculating a first output of a convolution layer using input activation and weights at step t;
calculating a second output of the convolution layer using the input activation and the weights at step (t−1); and
measuring a difference between the first output and the second output to compute the AQS value of the convolution layer.

### Claim 10 (depends on 1)

The method of claim 1, wherein calculating the AQS value further comprises:
calculating a first output of a convolution layer using input activation and weights at a first data precision;
calculating a second output of the convolution layer using the input activation and the weights at a second data precision, the second data precision using a larger bit-width than the first data precision; and
measuring a difference between the first output and the second output to compute the AQS value of the convolution layer.

### Claim 11 (depends on 1)

The method of claim 1, wherein grouping the one or more convolution layers into the quantization group further comprises:
grouping operation (OP) layers that share a same input activation.

### Claim 12

A system operative to determine bit-widths for mixed-precision neural network computing on a target hardware platform, comprising:
memory to store a neural network; and
processing circuitry coupled to the memory and operative to:
calculate an activation quantization sensitivity (AQS) value for each of a plurality of convolution layers in the neural network, wherein the AQS value indicates sensitivity of convolution output to quantized convolution input;
form a plurality of quantization groups by grouping one or more convolution layers into a quantization group to be executed by a corresponding set of target hardware;
calculate a group AQS value for each quantization group based on AQS values of the convolution layers in the quantization group; and
select bit-widths supported by the target hardware platform for corresponding quantization groups to optimize, under a given constraint, a sensitivity metric that is calculated based on each quantization group's group AQS value.

### Claim 13 (depends on 12)

The system of claim 12, wherein the processing circuitry is further operative to:
perform Integer Linear Programming (ILP) to obtain the bit-width for each quantization group that optimizes a total AQS under the given constraint, wherein the total AQS is a sum of all group AQS values in the neural network.

### Claim 14 (depends on 12)

The system of claim 12, the given constraint is total binary operations (BOPS), which limits a sum of per-group BOPS, and wherein per-group BOPS is calculated as the number of MAC operations multiplied by the bit-width of input activation and the bit-width of weights.

### Claim 15 (depends on 12)

The system of claim 12, wherein the processing circuitry is further operative to:
select an input activation bit-width and a weight bit-width for each quantization group to optimize the sensitivity metric under the given constraint.

### Claim 16 (depends on 12)

The system of claim 12, wherein the processing circuitry is further operative to:
identify one or more quantization groups to lower corresponding bit-widths under a user-defined accuracy criterion.

### Claim 17 (depends on 16)

The system of claim 16, wherein an identified quantization group is one having a highest group quality
log
⁢
(
MAC
+
1
)
goup
⁢
AQS
among all quantization groups, wherein MAC represents a number of multiply-and-add operations.

### Claim 18 (depends on 12)

The system of claim 12, wherein the group AQS value is the maximum AQS value of all convolution layers in the quantization group.

### Claim 19 (depends on 12)

The system of claim 12, wherein the processing circuitry when calculating the AQS value is further operative to:
calculate a first output of a convolution layer using input activation and weights at step t;
calculate a second output of the convolution layer using the input activation and the weights at step (t−1); and
measure a difference between the first output and the second output to compute the AQS value of the convolution layer.

### Claim 20 (depends on 12)

The system of claim 12, wherein the processing circuitry when calculating the AQS value is further operative to:
calculate a first output of a convolution layer using input activation and weights at a first data precision;
calculate a second output of the convolution layer using the input activation and the weights at a second data precision, the second data precision using a larger bit-width than the first data precision; and
measure a difference between the first output and the second output to compute the AQS value of the convolution layer.
