# US 2020/0264876 A1 · Adjusting activation compression for neural network training

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0264876 A1
- Document ID: US20200264876A1
- Status: Published application
- Kind: A1
- Application number: 16/276,395
- Filed: 2019-02-14
- Published: 2020-08-20
- Earliest claimed: 2019-02-14
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Daniel Lo; Bita Darvish Rouhani; Eric S. Chung; Yiren Zhao; Amar Phanishayee; Ritchie Zhao
- Web page: https://patentgrep.com/patent/US20200264876A1

## CPC

- G06N 3/063: using electronic means
- G06F 9/30025: {Format conversion instructions, e.g. Floating-Point to Integer, decimal conversion}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/048: Activation functions
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and, in particular, for adjusting floating-point formats used to store activation values during training. In certain examples of the disclosed technology, a computing system includes processors, memory, and a floating-point compressor in communication with the memory. The computing system is configured to produce a neural network comprising activation values expressed in a first floating-point format, select a second floating-point format for the neural network based on a performance metric, convert at least one of the activation values to the second floating-point format, and store the compressed activation values in the memory. Aspects of the second floating-point format that can be adjusted include the number of bits used to express mantissas, exponent format, use of non-uniform mantissas, and/or use of outlier values to express some of the mantissas.

## Claims

### Claim 1

A computing system comprising:
one or more processors;
bulk memory comprising computer-readable storage devices and/or memory;
a floating-point compressor formed from at least one of the processors, the floating-point compressor being in communication with the bulk memory; and
the computing system being configured to:
produce a neural network comprising activation values expressed in a first floating-point format,
select a second floating-point format for the neural network based on a performance metric for the neural network,
convert at least one of the activation values to the second floating-point format, thereby producing compressed activation values, and
with at least one of the processors, store the compressed activation values in the bulk memory.

### Claim 2 (depends on 1)

The computing system of claim 1, being further configured to:
convert at least one of the compressed activation values to the first floating-point format to produce uncompressed activation values; and
perform backward propagation for at least one layer of the neural network with the uncompressed activation values, producing a further trained neural network.

### Claim 3 (depends on 2)

The computing system of claim 2, wherein the computing system is further configured to:
perform forward propagation for the further trained neural network, producing updated activation values;
based on the updated activation values, determine an updated performance metric;
based on the updated performance metric, select a third floating-point format different than the second floating-point format;
converting at least one of the updated activation values to the third floating-point format to produce second compressed activation values; and
store the second compressed activation values in the bulk memory.

### Claim 4 (depends on 1)

The computing system of claim 1, wherein the second floating-point format has at least one mantissa expressed in an outlier mantissa format or a non-uniform mantissa format.

### Claim 5 (depends on 1)

The computing system of claim 1, being further configured to:
determine differences between an output of a layer of the neural network from an expected output; and
based on the determined differences, select a third floating-point format by increasing a number of mantissa bits used to store the compressed activation values.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein the compressor is further configured to further compress the compressed activation values prior to the storing by performing at least one or more of the following: entropy compression, zero compression, run length encoding, compressed sparse row compression, or compressed sparse column compression.

### Claim 7 (depends on 1)

The computing system of claim 1, wherein:
the processors comprise at least one of the following: a tensor processing unit, a neural network accelerator, a graphics processing unit, or a processor implemented in a reconfigurable logic array;
the bulk memory is situated on a different integrated circuit than the processors; and
wherein the bulk memory includes dynamic random access memory (DRAM) or embedded DRAM and the system further comprises a hardware accelerator including a memory temporarily storing the first activation values for at least a portion of only one layer of the neural network, the hardware accelerator memory including static RAM (SRAM) or a register file.

### Claim 8

A method of operating a computing system implementing a neural network, the method comprising:
with the computing system:
selecting an activation compression format based on a training performance metric for the neural network; and
storing compressed activation values for the neural network expressed in the activation compression format in a computer-readable memory or storage device.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
producing the training performance metric by calculating accuracy or change in accuracy of at least one layer of the neural network.

### Claim 10 (depends on 8)

The method of claim 8, further comprising:
uncompressing the stored compressed activation values and training the neural network;
evaluating the training performance metric for the trained neural network;
based on the training performance metric, adjusting the activation compression format to an increased precision; and
storing activation values for at least one layer of the trained neural network in the adjusted activation compression form.

### Claim 11 (depends on 10)

The method of claim 10, wherein:
the adjusted activation compression format has a greater number of mantissa bits than the selected activation compression format; and/or
the selected activation compression format is a block floating-point format and the adjusted activation compression format is a normal-precision floating point format.

### Claim 12 (depends on 10)

The method of claim 10, wherein the selected activation compression format comprises an outlier mantissa and the adjusted activation compression format does not comprise an outlier mantissa.

### Claim 13 (depends on 10)

The method of claim 10, wherein the selected activation compression format comprises a non-uniform mantissa.

### Claim 14 (depends on 8)

The method of claim 8, wherein the training performance metric is based at least in part on one or more of the following for the neural network: a true positive rate, a true negative rate, a positive predictive rate, a negative predictive value, a false negative rate, a false positive rate, a false discovery rate, a false omission rate, or an accuracy rate.

### Claim 15 (depends on 8)

The method of claim 8, wherein the training performance metric is based at least in part on one or more of the following for at least one layer of the neural network: mean square error, perplexity, gradient signal to noise ratio, or entropy.

### Claim 16 (depends on 8)

A quantization-enabled system comprising:
at least one processor;
a special-purpose processor configured to implement neural network operations in quantized number formats;
a bulk memory configured to received compressed activation values from the special-purpose processor; and
one or more computer-readable storage devices or media storing computer-executable instructions, which when executed by a computer, cause the computer to perform the method of claim 8.

### Claim 17

A quantization-enabled system comprising:
means for compressing neural network activation values produced during neural network training according to an accuracy parameter;
means for storing the compressed neural network activation values; and
means for evaluating the neural network during the neural network training and, based on the evaluating, adjusting the accuracy parameter.

### Claim 18 (depends on 17)

The quantization-enabled system of claim 17, wherein the means for compressing comprises:
means for converting activation values from a first floating-point format to a second-floating-point format according to the accuracy parameter.

### Claim 19 (depends on 17)

The quantization-enabled system of claim 17, wherein the means for compressing comprises:
means for expressing the compressed neural network activation values in a lossy mantissa format; and/or
means for expressing the compressed neural network activation values in an outlier-quantized format.

### Claim 20 (depends on 17)

The quantization-enabled system of claim 17, wherein the means for evaluating comprises at least one of the following:
means for evaluating accuracy of a neural network;
means for evaluating mean square error of the neural network;
means for evaluating perplexity of the neural network;
means for evaluating gradient signal to noise ratio of the neural network; and/or
means for evaluating entropy of the neural network.
