# US 2021/0174172 A1 · Method for automatic hybrid quantization of deep artificial neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174172 A1
- Document ID: US20210174172A1
- Status: Published application
- Kind: A1
- Application number: 17/112,889
- Filed: 2020-12-04
- Published: 2021-06-10
- Earliest claimed: 2019-12-04
- Assignees: Deep Vision Inc.
- Applicants: Deep Vision Inc.
- Inventors: Wajahat Qadeer; Rehan Hameed; Satyanarayana Raju Uppalapati; Abhilash Bharath Ghanore; Kasanagottu Sai Ram
- Web page: https://patentgrep.com/patent/US20210174172A1

## CPC

- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/045: Combinations of networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method includes, for each floating-point layer in a set of floating-point layers: calculating a set of input activations and a set of output activations of the floating-point layer; converting the floating-point layer to a low-bit-width layer; calculating a set of low-bit-width output activations based on the set of input activations; and calculating a per-layer deviation statistic of the low-bit-width layer. The method also includes ordering the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer. The method additionally includes, while a loss-of-accuracy threshold exceeds the accuracy of the quantized network: converting a floating-point layer represented by the low-bit-width layer to a high-bit-width layer; replacing the low-bit-width layer with the high-bit-width layer in the quantized network; updating the accuracy of the quantized network; and, in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

## Claims

### Claim 1

A method for quantizing artificial neural networks including:
accessing a floating-point network comprising a set of floating-point layers;
accessing a set of validation examples for the floating-point network;
for each floating-point layer in the set of floating-point layers:
calculating a set of example input activations of the floating-point layer based on the set of validation examples and a preceding subset of floating-point layers in the floating-point network;
calculating a set of example output activations of the floating-point layer based on the set of example input activations of the floating point layer;
converting the floating-point layer to a low-bit-width layer in a set of low-bit-width layers;
calculating a set of low-bit-width output activations of the low-bit-width layer based on the set of example input activations; and
calculating a per-layer deviation statistic of the low-bit-width layer based on the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer;
generating a quantized network representing the floating-point network and comprising the set of low-bit-width layers;
calculating an accuracy of the quantized network based on the set of validation examples; and
while a loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, from a greatest-deviating low-bit-width layer in the set of low-bit-width layers toward a least-deviating low-bit-width layer in the set of low-bit-width layers:
converting a floating-point layer represented by the low-bit-width layer to a high-bit-width layer;
replacing the low-bit-width layer with the high-bit-width layer in the quantized network;
updating the accuracy of the quantized network based on the set of validation examples; and
in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

### Claim 2 (depends on 1)

The method of claim 1:
wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises converting the floating-point layer to an eight-bit layer in a set of eight-bit layers;
wherein calculating the set of low-bit-width output activations of the low-bit-width layer based on the set of example input activations comprises calculating a set of eight-bit output activations of the eight-bit layer based on the set of example input activations;
wherein calculating the per-layer deviation statistic of the low-bit-width layer based on the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer comprises calculating a per-layer deviation statistic of the eight-bit layer based on the set of eight-bit output activations of the eight-bit layer and the set of example output activations of the floating-point layer;
wherein generating the quantized network representing the floating-point network and comprising the set of low-bit-width layers comprises generating the quantized network representing the floating-point network and comprising the set of eight-bit layers; and
further comprising, while the loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, from a greatest-deviating eight-bit layer in the set of eight-bit layers toward a least-deviating eight-bit layer in the set of eight-bit layers:
converting a floating-point layer represented by the eight-bit layer to a sixteen-bit layer;
replacing the eight-bit layer with the sixteen-bit layer in the quantized network;
updating the accuracy of the quantized network based on the set of validation examples; and
in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

### Claim 3 (depends on 1)

The method of claim 1, further comprising, in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, loading the quantized network onto an edge device.

### Claim 4

A method for quantizing artificial neural networks including:
accessing a floating-point network comprising a set of floating-point layers;
accessing a set of validation examples for the floating-point network;
for each floating-point layer in the set of floating-point layers:
calculating a set of example input activations of the floating-point layer and a set of example output activations of the floating-point layer based on the set of validation examples;
converting the floating-point layer to a low-bit-width layer in a set of low-bit-width layers;
calculating a set of low-bit-width output activations of the low-bit-width layer based on the set of example input activations; and
calculating a per-layer deviation statistic of the low-bit-width layer based on the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer;
generating a quantized network representing the floating-point network and comprising the set of low-bit-width layers;
calculating an accuracy of the quantized network based on the set of validation examples;
ordering the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers to generate an ordered set of low-bit-width layers; and
while a loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, in the ordered set of low-bit-width layers:
converting a floating-point layer represented by the low-bit-width layer to a high-bit-width layer;
replacing the low-bit-width layer with the high-bit-width layer in the quantized network;
updating the accuracy of the quantized network based on the set of validation examples; and
in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

### Claim 5 (depends on 4)

The method of claim 4:
wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises converting the floating-point layer to an eight-bit layer in a set of eight-bit layers;
wherein calculating the set of low-bit-width output activations of the low-bit-width layer based on the set of example input activations comprises calculating a set of eight-bit output activations of the eight-bit layer based on the set of example input activations;
wherein calculating the per-layer deviation statistic of the low-bit-width layer based on the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer comprises calculating a per-layer deviation statistic of the eight-bit layer based on the set of eight-bit output activations of the eight-bit layer and the set of example output activations of the floating-point layer;
wherein generating the quantized network representing the floating-point network and comprising the set of low-bit-width layers comprises generating the quantized network representing the floating-point network and comprising the set of eight-bit layers;
wherein ordering the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers to generate the ordered set of low-bit-width layers comprises ordering the set of eight-bit layers based on the per-layer deviation statistic of each eight-bit layer in the set of eight-bit layers to generate an ordered set of eight-bit layers; and
further comprising, while the loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, in the ordered set of eight-bit layers:
converting a floating-point layer represented by the eight-bit layer to a sixteen-bit layer;
replacing the eight-bit layer with the sixteen-bit layer in the quantized network;
updating the accuracy of the quantized network based on the set of validation examples; and
in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

### Claim 6 (depends on 4)

The method of claim 4, wherein ordering the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers to generate the ordered set of low-bit-width layers comprises ordering the set of low-bit-width layers, based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers and a network position of each low-bit-width layer in the set of low-bit-width layers, to generate the ordered set of low-bit-width layers.

### Claim 7 (depends on 6)

The method of claim 6, wherein ordering the set of low-bit-width layers, based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers and the network position of each low-bit-width layer in the set of low-bit-width layers, to generate the ordered set of low-bit-width layers comprises:
selecting a highest-deviating subset of the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers; and
ordering the highest-deviating subset from an earliest network position to a latest network position based on the network position of each low-bit-width layer in the set of low-bit-width layers to generate the ordered set of low-bit-width layers.

### Claim 8 (depends on 4)

The method of claim 4, wherein ordering the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers to generate the ordered set of low-bit-width layers comprises ordering the set of low-bit-width layers, based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers and a layer type of each low-bit-width layer in the set of low-bit-width layers, to generate the ordered set of low-bit-width layers.

### Claim 9 (depends on 4)

The method of claim 4, wherein ordering the set of low-bit-width layers based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers to generate the ordered set of low-bit-width layers comprises ordering the set of low-bit-width layers, based on the per-layer deviation statistic of each low-bit-width layer in the set of low-bit-width layers and a saturation proportion of each low-bit-width layer in the set of low-bit-width layers, to generate the ordered set of low-bit-width layers.

### Claim 10 (depends on 4)

The method of claim 4, wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises, for each floating-point channel of the floating-point layer, converting the floating-point channel to a low-bit-width channel in the low-bit-width layer in the set of low-bit-width layers.

### Claim 11 (depends on 4)

The method of claim 4, wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises, for each floating-point output channel of the floating-point layer, converting the floating-point output channel to a low-bit-width output channel in the low-bit-width layer in the set of low-bit-width layers.

### Claim 12 (depends on 4)

The method of claim 4, wherein calculating the per-layer deviation statistic of the low-bit-width layer based on the set of low-bit-width output activations of the low-bit-width layer and the set of example output activations of the floating-point layer comprises:
for each example output activation in the set of example output activations, calculating an error metric between a corresponding low-bit-width output activation in the set of low-bit-width output activations and the example output activation; and
calculating the per-layer deviation statistic equal to a mean squared error of the error metric for each example output activation in the set of example output activations.

### Claim 13 (depends on 4)

The method of claim 4, wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers, the low-bit-width layer represented in Q-format fixed-point notation.

### Claim 14 (depends on 13)

The method of claim 13, wherein converting the floating-point layer to the low-bit-width layer in the set of low-bit-width layers comprises:
converting a set of floating-point weights of the floating-point layer to a set of Q-format fixed-point weights of the low-bit-width layer in the set of low-bit-width layers;
converting a bias of the floating-point layer to a Q-format fixed-point bias of the low-bit-width layer in the set of low-bit-width layers; and
bit-shifting the set of Q-format fixed-point weights to match the Q-format fixed-point bias.

### Claim 15 (depends on 4)

The method of claim 4, further comprising, in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, loading the quantized network onto an edge device.

### Claim 16 (depends on 4)

The method of claim 4, further comprising:
calculating a performance metric of the quantized network based on the set of validation examples; and
while the loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, in the ordered set of low-bit-width layers, updating the performance metric of the quantized network based on the set of validation examples.

### Claim 17 (depends on 16)

The method of claim 16, further comprising, while the loss-of-accuracy threshold exceeds the accuracy of the quantized network and while the performance metric of the quantized network exceeds a performance metric threshold, sequentially, in the ordered set of low-bit-width layers:
converting the floating-point layer represented by the low-bit-width layer to the high-bit-width layer;
replacing the low-bit-width layer with the high-bit-width layer in the quantized network;
updating the accuracy of the quantized network based on the set of validation examples;
updating the performance metric of the quantized network based on the set of validation examples; and
in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

### Claim 18 (depends on 16)

The method of claim 16, further comprising rendering a plot of the accuracy of the quantized network and the performance metric of the quantized network for each replacement of a low-bit-width layer with a high-bit-width layer in the quantized network.

### Claim 19 (depends on 4)

The method of claim 4, further comprising, while the loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, from a greatest-deviating low-bit-width layer in the set of low-bit-width layers toward a least-deviating low-bit-width layer in the set of low-bit-width layers:
converting the floating-point layer represented by the low-bit-width layer to the high-bit-width layer;
replacing the low-bit-width layer with the high-bit-width layer in the quantized network;
updating the accuracy of the quantized network based on the set of validation examples; and
in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, returning the quantized network.

### Claim 20

A method for quantizing artificial neural networks including:
accessing a floating-point network comprising a set of floating-point layers;
accessing a set of validation examples for the floating-point network;
for each floating-point layer in the set of floating-point layers:
calculating a set of example input activations of the floating-point layer and a set of example output activations of the floating-point layer based on the set of validation examples;
converting the floating-point layer to an eight-bit layer in a set of eight-bit layers;
calculating a set of eight-bit output activations of the eight-bit layer based on the set of example input activations; and
calculating a per-layer deviation statistic of the eight-bit layer based on the eight-bit output activations of the eight-bit layer and the set of example output activations of the floating-point layer;
generating a quantized network representing the floating-point network and comprising the set of eight-bit layers;
calculating an accuracy of the quantized network based on the set of validation examples;
ordering the set of eight-bit layers based on the per-layer deviation statistic of each eight-bit layer in the set of eight-bit layers to generate an ordered set of eight-bit layers; and
while a loss-of-accuracy threshold exceeds the accuracy of the quantized network, sequentially, in the ordered set of eight-bit layers:
converting a floating-point layer represented by the eight-bit layer to a sixteen-bit layer;
replacing the eight-bit layer with the sixteen-bit layer in the quantized network;
updating the accuracy of the quantized network based on the set of validation examples; and
in response to the accuracy of the quantized network exceeding the loss-of-accuracy threshold, loading the quantized network onto an edge device.
