# US 2022/0067512 A1 · Fine-grained per-vector scaling for neural network quantization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0067512 A1
- Document ID: US20220067512A1
- Status: Published application
- Kind: A1
- Application number: 17/086,114
- Filed: 2020-10-30
- Published: 2022-03-03
- Earliest claimed: 2020-08-28
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Brucek Kurdo Khailany; Steve Haihang Dai; Rangharajan Venkatesan; Haoxing Ren
- Web page: https://patentgrep.com/patent/US20220067512A1

## CPC

- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06F 2207/4824: Neural networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Today neural networks are used to enable autonomous vehicles and improve the quality of speech recognition, real-time language translation, and online search optimizations. However, operation of the neural networks for these applications consumes energy. Quantization of parameters used by the neural networks reduces the amount of memory needed to store the parameters while also reducing the power consumed during operation of the neural network. Matrix operations performed by the neural networks require many multiplication calculations, so reducing the number of bits that are multiplied reduces the energy that is consumed. Quantizing smaller sets of the parameters using a shared scale factor improves accuracy compared with quantizing larger sets of the parameters. Accuracy of the calculations may be maintained by quantizing and scaling the parameters using fine-grained per-vector scale factors. A vector includes one or more elements within a single dimension of a multi-dimensional matrix.

## Claims

### Claim 1

A computer-implemented method for quantizing parameters of a neural network model, the method comprising:
computing, for each vector of parameters within a single dimension of a multi-dimensional parameter tensor, a scale factor to produce a plurality of scale factors for the multi-dimensional parameter tensor;
reducing, for each vector of parameters, a precision of each parameter in the vector of parameters according to a corresponding scale factor to generate a reduced precision vector of parameters; and
processing the reduced precision vectors of parameters by a layer of the neural network model to produce output values.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the parameters are at least one of weights or activations.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the parameters are weights and the scale factors are precalculated based on the weights.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the parameters are calculated by one of post-training calibration or quantization-aware retraining.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the parameters are activations and the scale factors are dynamically calculated at each layer of the neural network model during inferencing.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the processing comprises:
computing vector dot-products for each vector of the parameters;
modifying the vector dot-products according to the scale factor corresponding to each vector of the parameters to produce scaled vector dot-products for each vector of the parameters; and
accumulating the scaled vector dot-products for each vector of the parameters to produce elements of a multi-dimensional output tensor as the output values.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the parameters comprise weights and a second multi-dimensional parameter tensor comprises activations, and wherein modifying the vector dot-products comprises multiplying each vector dot-product by a corresponding scale factor for the weights and a corresponding scale factor for the activations used to compute the vector dot-product.

### Claim 8 (depends on 6)

The computer-implemented method of claim 6, wherein the parameters comprise weights and a second multi-dimensional parameter tensor comprises activations, and wherein modifying the vector dot-products comprises:
multiplying a corresponding scale factor for the weights and a corresponding scale factor for the activations used to compute the vector dot-product to produce a scale product factor;
rounding the scale product factor; and
multiplying each vector dot-product by the rounded scale factor to produce the scaled vector dot-products.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising computing a coarse scale factor for the multi-dimensional parameter tensor, wherein the coarse scale factor is one of a per-layer, per-tensor, or per-channel scale factor.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, further comprising modifying the output values by multiplying each output value by the coarse scale factor.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9, wherein the scale factor is an integer format number and the coarse scale factor is a floating-point format number.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein the steps of computing, reducing, and processing are performed on a server or in a data center to generate data, and the data are streamed to a user device.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the steps of computing, reducing, and processing are performed within a cloud computing environment.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein the steps of computing, reducing, and processing generate data that are used for training, testing, or certifying the neural network model or another neural network model that is employed in a machine, robot, or autonomous vehicle.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, wherein the steps of computing, reducing, and processing are performed on a virtual machine comprising a portion of a graphics processing unit or multi-core processor.

### Claim 16

A system, comprising:
a processor configured to implement a neural network model that is configured to quantize parameters by:
computing, for each vector of parameters within a single dimension of a multi-dimensional parameter tensor, a scale factor to produce a plurality of scale factors for the multi-dimensional parameter tensor;
reducing, for each vector of parameters, a precision of each parameter in the vector of parameters according to a corresponding scale factor to generate a reduced precision vector of parameters; and
processing the reduced precision vectors of parameters by a layer of the neural network model to produce output values.

### Claim 17 (depends on 16)

The system of claim 16, wherein the parameters are activations and the scale factors are dynamically calculated at each layer of the neural network model during inferencing.

### Claim 18 (depends on 16)

The system of claim 16, wherein the processing comprises:
computing vector dot-products for each vector of the parameters;
modifying the vector dot-products according to the scale factor corresponding to each vector of the parameters to produce scaled vector dot-products for each vector of the parameters; and
accumulating the scaled vector dot-products for each vector of the parameters to produce elements of a multi-dimensional output tensor as the output values.

### Claim 19

A non-transitory computer-readable media storing computer instructions for quantizing parameters that, when executed by one or more processors, cause the one or more processors to perform the steps of:
computing, for each vector of parameters within a single dimension of a multi-dimensional parameter tensor, a scale factor to produce a plurality of scale factors for the multi-dimensional parameter tensor;
reducing, for each vector of parameters, a precision of each parameter in the vector of parameters according to a corresponding scale factor to generate a reduced precision vector of parameters; and
processing the reduced precision vectors of parameters by a layer of a neural network model to produce output values.

### Claim 20 (depends on 19)

The non-transitory computer-readable media of claim 19, wherein the parameters are activations and the scale factors are dynamically calculated at each layer of the neural network model during inferencing.

### Claim 21 (depends on 19)

The non-transitory computer-readable media of claim 19, wherein the processing comprises:
computing vector dot-products for each vector of the parameters;
modifying the vector dot-products according to the scale factor corresponding to each vector of the parameters to produce scaled vector dot-products for each vector of the parameters; and
accumulating the scaled vector dot-products for each vector of the parameters to produce elements of a multi-dimensional output tensor as the output values.
