# US 2025/0061320 A1 · Adjusting activation compression for neural network training

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0061320 A1
- Document ID: US20250061320A1
- Status: Published application
- Kind: A1
- Application number: 18/936,699
- Filed: 2024-11-04
- Published: 2025-02-20
- Earliest claimed: 2019-02-14
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Daniel Lo; Bita Darvish Rouhani; Eric S. Chung; Yiren Zhao; Amar Phanishayee; Ritchie Zhao
- Web page: https://patentgrep.com/patent/US20250061320A1

## CPC

- G06N 3/063: using electronic means
- G06F 9/30025: {Format conversion instructions, e.g. Floating-Point to Integer, decimal conversion}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/048: Activation functions
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Apparatus and methods for training a neural network accelerator using quantized precision data formats are disclosed, and, in particular, for adjusting floating-point formats used to store activation values during training. In certain examples of the disclosed technology, a computing system includes processors, memory, and a floating-point compressor in communication with the memory. The computing system is configured to produce a neural network comprising activation values expressed in a first floating-point format, select a second floating-point format for the neural network based on a performance metric, convert at least one of the activation values to the second floating-point format, and store the compressed activation values in the memory. Aspects of the second floating-point format that can be adjusted include the number of bits used to express mantissas, exponent format, use of non-uniform mantissas, and/or use of outlier values to express some of the mantissas.

## Claims

### Claim 1

A computing system comprising:
one or more processors;
bulk memory comprising computer-readable storage devices and/or memory;
a floating-point compressor formed from at least one of the processors, the floating-point compressor being in communication with the bulk memory; and
the computing system being configured to:
produce a neural network comprising activation values expressed in a first floating-point format;
select a second floating-point format for the neural network based on a value of a performance metric for the neural network;
convert at least one of the activation values to the second floating-point format, thereby producing compressed activation values; and
with at least one of the processors, store the compressed activation values in the bulk memory.

### Claim 2 (depends on 1)

The computing system of claim 1, being further configured to:
convert at least one of the compressed activation values to the first floating-point format to produce uncompressed activation values; and
perform backward propagation for at least one layer of the neural network with the uncompressed activation values, producing a further trained neural network.

### Claim 3 (depends on 2)

The computing system of claim 2, wherein the computing system is further configured to:
perform forward propagation for the further trained neural network, producing updated activation values;
based on the updated activation values, determine an updated performance metric value;
based on the updated performance metric value, select a third floating-point format different than the second floating-point format;
convert at least one of the updated activation values to the third floating-point format to produce second compressed activation values; and
store the second compressed activation values in the bulk memory.

### Claim 4 (depends on 1)

The computing system of claim 1, wherein the second floating-point format has at least one mantissa expressed in an outlier mantissa format or a non-uniform mantissa format.

### Claim 5 (depends on 1)

The computing system of claim 1, being further configured to:
determine differences between an output of a layer of the neural network from an expected output; and
based on the determined differences, select a third floating-point format by increasing a number of mantissa bits used to store the compressed activation values.

### Claim 6 (depends on 1)

The computing system of claim 1, wherein the floating-point compressor is further configured to further compress the compressed activation values prior to the storing by performing at least one or more of the following: entropy compression, zero compression, run length encoding, compressed sparse row compression, or compressed sparse column compression.

### Claim 7 (depends on 1)

The computing system of claim 1, wherein:
the processors comprise at least one of the following: a tensor processing unit, a neural network accelerator, a graphics processing unit, or a processor implemented in a reconfigurable logic array;
the bulk memory is situated on a different integrated circuit than the processors; and
the bulk memory includes dynamic random access memory (DRAM) or embedded DRAM and the system further comprises a hardware accelerator including a memory temporarily storing the first activation values for at least a portion of only one layer of the neural network, the hardware accelerator memory including static RAM (SRAM) or a register file.

### Claim 8

A quantization-enabled system comprising:
means for compressing neural network activation values produced during neural network training using a floating-point format selected based on a value of a performance metric;
means for storing the compressed neural network activation values produced after selection of the floating-point format;
means for evaluating the performance metric of the neural network during training; and
means for adjusting the floating-point format based on evaluation of the performance metric to maintain or optimize neural network performance.

### Claim 9 (depends on 8)

The quantization-enabled system of claim 8, wherein the means for compressing comprises:
means for converting activation values from a first floating-point format to a second-floating-point format according to the value of the performance metric.

### Claim 10 (depends on 8)

The quantization-enabled system of claim 8, wherein the means for compressing neural network activation values comprises:
means for expressing the compressed neural network activation values in a lossy mantissa format; and/or
means for expressing the compressed neural network activation values in an outlier-quantized format.

### Claim 11 (depends on 8)

The quantization-enabled system of claim 8, wherein the means for evaluating comprises at least one of the following:
means for evaluating accuracy of a neural network;
means for evaluating mean square error of the neural network;
means for evaluating perplexity of the neural network;
means for evaluating gradient signal to noise ratio of the neural network; and/or
means for evaluating entropy of the neural network.

### Claim 12 (depends on 8)

The quantization-enabled system of claim 8, wherein the means for compressing is further configured to compress the compressed activation values prior to storing by performing at least one of the following: entropy compression, zero compression, run-length encoding, compressed sparse row compression, or compressed sparse column compression.

### Claim 13 (depends on 8)

The quantization-enabled system of claim 8, further comprising:
means for performing forward propagation for a further trained neural network to produce updated activation values;
means for determining an updated value of the performance metric based on the updated activation values;
means for selecting a third floating-point format different from the second floating-point format based on the updated value of the performance metric;
means for converting at least one of the updated activation values to the third floating-point format to produce second compressed activation values; and
means for storing the second compressed activation values in memory.

### Claim 14

A method, implemented in a computing system comprising one or more processors, bulk memory comprising computer-readable storage devices and/or memory, and a floating-point compressor formed from at least one of the processors, the floating-point compressor being in communication with the bulk memory, the method comprising:
producing a neural network comprising activation values expressed in a first floating-point format,
selecting a second floating-point format for the neural network based on a value of a performance metric for the neural network,
converting at least one of the activation values to the second floating-point format, thereby producing compressed activation values, and
with at least one of the processors, storing the compressed activation values in the bulk memory.

### Claim 15 (depends on 14)

The method of claim 14, further comprising:
converting at least one of the compressed activation values to the first floating-point format to produce uncompressed activation values; and
performing backward propagation for at least one layer of the neural network with the uncompressed activation values, producing a further trained neural network.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
performing forward propagation for the further trained neural network, producing updated activation values;
based on the updated activation values, determining an updated performance metric value;
based on the updated performance metric value, selecting a third floating-point format different than the second floating-point format;
converting at least one of the updated activation values to the third floating-point format to produce second compressed activation values; and
storing the second compressed activation values in the bulk memory.

### Claim 17 (depends on 14)

The method of claim 14, wherein the second floating-point format has at least one mantissa expressed in an outlier mantissa format or a non-uniform mantissa format.

### Claim 18 (depends on 14)

The method of claim 14, further comprising:
determining differences between an output of a layer of the neural network from an expected output; and
based on the determined differences, selecting a third floating-point format by increasing a number of mantissa bits used to store the compressed activation values.

### Claim 19 (depends on 14)

The method of claim 14, further comprising
by the floating-point compressor, further compressing the compressed activation values prior to the storing by performing at least one or more of the following: entropy compression, zero compression, run length encoding, compressed sparse row compression, or compressed sparse column compression.

### Claim 20 (depends on 14)

The method of claim 14, wherein the storing of the compressed activation values is performed in bulk memory situated on a different integrated circuit than the processors, wherein the bulk memory comprises dynamic random access memory (DRAM) or embedded DRAM, and wherein the method further comprises:
temporarily storing the activation values for at least a portion of only one layer of the neural network in a hardware accelerator memory, wherein the hardware accelerator memory comprises static RAM (SRAM) or a register file.
