US 11,537,853 B1Granted

Decompression and compression of neural network data using different compression schemes

AssigneeAmazon Technologies, Inc.

InventorsTariq Afzal, Arvind Mandhani

Life of this patent

Estimating the term…

Loading patent…

Abstract

Described herein is a neural network accelerator (NNA) with a decompression unit that can be configured to perform multiple types of decompression. The decompression may include a separate subunit for each decompression type. The subunits can be coupled to form a pipeline in which partially decompressed results generated by one subunit are input for further decompression by another subunit. Depending on which types of compression were applied to incoming data, any number of the subunits may be used to produce a decompressed output. In some embodiments, the decompression unit is configured to decompress data that has been compressed using a zero value compression scheme, a shared value compression scheme, or both. The NNA can also include a compression unit implemented in a manner similar to that of the decompression unit.

Claims20

Dependent claims

A computing system, comprising:

a first memory storing compressed data, wherein the compressed data corresponds to original data that was compressed by applying a zero value compression scheme in combination with a shared value compression scheme, the original data comprising a first data value, a second data value, and a third data value, wherein the first data value, the second data value, and the third data value each represent a different weight of a neural network,
wherein the zero value compression scheme involves removal of the first data value based on the first data value being within a first range around zero, and
wherein the shared value compression scheme involves:
assigning the second data value and the third data value to a first cluster in a set of clusters based on proximity of the second data value and the third data value to a value associated with the first cluster,
substituting a binary-encoded value for the second data value, the binary-encoded value being an index to the first cluster, and
substituting the binary-encoded value for the third data value;
a second memory;
a data move engine configured to generate decompressed data by processing the compressed data through a first decompression unit and a second decompression unit,
wherein the first decompression unit is configured to introduce a zero value in place of the first data value by:
identifying a position of the first data value within the original data, the position of the first data value being indicated by a compression map that contains information on a position of any data value that was removed from the original data as a result of zero value compression, and
inserting, into the compressed data, a zero at the position of the first data value,
wherein the second decompression unit is configured to:
identify the value associated with the first cluster by referencing a lookup table that maps the index to the value associated with the first cluster, and
replace each instance of the binary-encoded value within the compressed data with the value associated with the first cluster, and
wherein the data move engine is further configured to store the decompressed data into the second memory;
and
a processing unit that performs a computation using the decompressed data stored in the second memory.

14 more elements

A computing system, comprising:

a host processor configured to generate first compressed data by compressing first data according to a first compression scheme;
and
a neural network processor configured to execute a neural network, the neural network processor comprising:
a memory;
a processing unit;
a first decompression unit operable to perform decompression in accordance with the first compression scheme, wherein the first decompression unit is configured to:
receive first compression information from the host processor or as part of the first compressed data;
and
generate first decompressed data by decompressing the first compressed data using the first compression information;
and
a second decompression unit configured to:
receive the first decompressed data from the first decompression unit;
and
send the first decompressed data to the memory or the processing unit;
wherein the memory is configured to:
store the first compressed data prior to decompression of the first compressed data by the first decompression unit;
or
store the first decompressed data;
and
wherein the processing unit generates inferences using the first decompressed data.

17 more elements

A method performed by a processor of a computing system, the method comprising:

generating, by the processor, first compressed data by compressing first data according to a first compression scheme, wherein the first data represents first weights of a neural network;
generating, by the processor, second compressed data by compressing the first data according to a second compression scheme;
determining, by the processor, a first performance metric, wherein the first performance metric represents an amount of compression associated with the first compressed data or an accuracy level of a first set of inferences produced by executing the neural network using the first compressed data;
determining, by the processor, a second performance metric, wherein the second performance metric represents an amount of compression associated with the second compressed data or an accuracy level of a second set of inferences produced by executing the neural network using the second compressed data;
determining that the first performance metric meets or exceeds a threshold corresponding to a minimum acceptable amount of compression or a minimum acceptable accuracy level;
and
based on determining that the first performance metric meets or exceeds the threshold, sending the first compressed data to a neural network processor of the computing system or sending the first compressed data over a network to a remote computing system.

4 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/455,258Filed 27 Jun 2019 · us provisional application
Priority: 28 Nov 2018
Granted: 27 Dec 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification28 codes

G10L 15/05
Segmentation; Word boundary detection: word boundary detection
G06F 5/01
Methods or arrangements for data conversion without changing the order or content of the data handled: for shifting, e.g. justifying, scaling,…
G06F 7/5443
For evaluating functions by calculation: sum of products
G06F 7/57
Using non-contact-making devices, e.g. tube, solid state device; using unspecified devices: arithmetic logic units [ALU], i.e. arrangements or…