# US 12,182,695 B1 · Fine-grained sparsity computations in systolic array

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,695 B1
- Document ID: US12182695B1
- Status: Granted
- Kind: B1
- Application number: 18/474,129
- Filed: 2023-09-25
- Granted: 2024-12-31
- Earliest claimed: 2020-06-30
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Paul Gilbert Meyer; Thiam Khean Hah; Randy Renfu Huang; Ron Diamant; Vignesh Vivekraja
- Web page: https://patentgrep.com/patent/US12182695B1

## CPC

- G06N 3/063: using electronic means
- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 9/3001: {Arithmetic instructions}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06F 9/3013: {according to data content, e.g. floating-point registers, address registers}
- G06F 9/3893: {controlled in tandem, e.g. multiplier-accumulator}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06F 2207/4824: Neural networks
- G06N 3/048: Activation functions
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A systolic array can implement an architecture tailored to perform matrix multiplications on sparse matrices. Each processing element in the systolic array may include a register configured to store a value, and a multiplexor configured to select an input element from multiple input data buses based on metadata associated with the value. Each processing element may also include a multiplier configured to multiply the selected input element with the value to generate a multiplication result, and an adder configured to add the multiplication result to a partial sum input to generate a partial sum output.

## Claims

### Claim 1

A neural network processor comprising:
a memory operable to store a constrained fine-grained sparse matrix as a set of non-zero values and metadata associated with each non-zero value indicating which row in the constrained fine-grained sparse matrix that the non-zero value corresponds to; and
an array of processing elements arranged in rows and columns, wherein each row of processing elements has a plurality of row input buses, wherein the array of processing elements is operable in a sparsity mode of operation and a normal mode of operation, and wherein the array of processing elements is operable to perform a sparse matrix multiplication operation on the constrained fine-grained sparse matrix by:
loading the set of non-zero values into a row of the array of processing elements; and
for each processing element in the row of the array of processing elements:
selecting an input element from the plurality of row input buses based on the metadata associated with the non-zero value loaded into the processing element;
multiplying the selected input element with the non-zero value loaded into the processing element to generate a multiplication result;
adding the multiplication result to a column input to generate a partial sum; and
outputting the partial sum to a column output.

### Claim 2 (depends on 1)

The neural network processor of claim 1, wherein the array of processing elements is operable to concurrently perform multiple sparse matrix multiplication operations using multiple rows of the array of processing elements.

### Claim 3 (depends on 1)

The neural network processor of claim 1, wherein the array of processing elements is operable to turn off unused rows of the array of processing elements while performing the sparse matrix multiplication operation.

### Claim 4 (depends on 1)

The neural network processor of claim 1, wherein the array of processing elements is operable to perform a matrix multiplication operation on a dense matrix using every row of the array of processing elements.

### Claim 5

An integrated circuit device comprising:
an array of processing elements arranged in rows and columns, wherein each processing element includes:
a register configured to store a value;
a multiplexor configured to select, based on metadata associated with the value stored in the register, an input element from a plurality of input data buses;
a multiplier configured to multiply the selected input element with the value stored in the register to generate a multiplication result; and
an adder configured to add the multiplication result to a partial sum input to generate a partial sum output,
wherein the array of processing elements is operable in a sparsity mode of operation and a normal mode of operation.

### Claim 6 (depends on 5)

The integrated circuit device of claim 5, wherein in the sparsity mode of operation, each row of the array of processing elements is operable to perform a sparse matrix multiplication operation on a two-dimensional sparse matrix.

### Claim 7 (depends on 5)

The integrated circuit device of claim 5, wherein in the sparsity mode of operation, the metadata associated with the value stored in the register includes sparsity row information to indicate which row of a two-dimensional sparse matrix that the value corresponds to, and the multiplexor is configured to select the input element based on the sparsity row information.

### Claim 8 (depends on 7)

The integrated circuit device of claim 7, wherein the sparsity row information is encoded using one-hot encoding.

### Claim 9 (depends on 7)

The integrated circuit device of claim 7, wherein the sparsity row information is represented as a row number.

### Claim 10 (depends on 5)

The integrated circuit device of claim 5, wherein in the normal mode of operation, the array of processing elements is operable to perform a matrix multiplication operation on a matrix using multiple rows of the array of processing elements.

### Claim 11 (depends on 5)

The integrated circuit device of claim 5, wherein in a normal mode of operation, the multiplexor is configured to select the input element on a predetermined input data bus.

### Claim 12 (depends on 5)

The integrated circuit device of claim 5, wherein the metadata includes mode information to control whether the array of processing elements is operating in the sparsity mode of operation or in the normal mode of operation.

### Claim 13

A method for operating an acceleration circuit having an array of processing elements arranged in rows and columns to perform a matrix multiplication operation, the method comprising:
receiving, by a processing element in the array of processing elements, a value for storing in a register;
storing, by the processing element, the value in the register;
receiving, by the processing element, a set of input elements on respective input data buses;
selecting, by a multiplexer of the processing element, an input element from the input data buses based on metadata associated with the value stored in the register;
multiplying, by a multiplier of the processing element, the selected input element with the value stored in the register to generate a multiplication result; and
adding, by an adder of the processing element, the multiplication result to a partial sum input to generate a partial sum output,
wherein the array of processing elements is operable in a sparsity mode of operations and a normal mode of operation.

### Claim 14 (depends on 13)

The method of claim 13, wherein the value stored in the register is a non-zero value of a sparse matrix loaded from a memory that stores the sparse matrix as a set of non-zero values.

### Claim 15 (depends on 14)

The method of claim 14, wherein the memory further stores sparsity row information associated with each non-zero value, the sparsity row information indicating which row in the sparse matrix that the non-zero value corresponds to.

### Claim 16 (depends on 15)

The method of claim 15, wherein the sparsity row information is represented using one-hot encoding or using a row number.

### Claim 17 (depends on 13)

The method of claim 13, wherein the matrix multiplication operation is a sparse matrix multiplication operation being performed on a sparse matrix, and the sparse matrix multiplication operation is performed using one row of the array of processing elements.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
turning off unused rows of the array of processing elements while performing the sparse matrix multiplication operation.

### Claim 19 (depends on 17)

The method of claim 17, further comprising:
performing another sparse matrix multiplication operation on another sparse matrix using another row of the array of processing elements concurrently with the sparse matrix multiplication operation.

### Claim 20 (depends on 17)

The method of claim 17, further comprising:
performing another matrix multiplication operation using every row of the array of processing elements.