# US 2021/0334142 A1 · Systolic array-friendly data placement and control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334142 A1
- Document ID: US20210334142A1
- Status: Published application
- Kind: A1
- Application number: 16/857,797
- Filed: 2020-04-24
- Published: 2021-10-28
- Earliest claimed: 2020-04-24
- Assignees: Alibaba Group Holding Limited
- Applicants: Alibaba Group Holding Limited
- Inventors: Yuhao Wang; Xiaoxin Fan; Dimin Niu; Chunsheng Liu; Wei Han
- Web page: https://patentgrep.com/patent/US20210334142A1

## CPC

- G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals}
- G06F 9/30134: {Register stacks; shift registers}
- G06F 9/3893: {controlled in tandem, e.g. multiplier-accumulator}
- G06F 9/4812: {by interrupt, e.g. masked}
- G06F 15/8046: {Systolic arrays}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

The present disclosure relates to an accelerator for systolic array-friendly data placement. The accelerator may include: a systolic array comprising a plurality of operation units, wherein the systolic array is configured to receive staged input data and perform operations using the staged input to generate staged output data, the staged output data comprising a number of segments; a controller configured to execute one or more instructions to generate a pattern generation signal; a data mask generator; and a memory configured to store the staged output data using the generated masks. The data mask generator may include circuitry configured to: receive the pattern generation signal from the controller, and, based on the received signal, generate a mask corresponding to each segment of the staged output data.

## Claims

### Claim 1

An accelerator comprising:
a systolic array comprising a plurality of operation units, wherein the systolic array is configured to receive staged input data and perform operations using the staged input to generate staged output data, the staged output data comprising a number of segments;
a controller configured to execute one or more instructions to generate a pattern generation signal;
a data mask generator including circuitry configured to:
receive the pattern generation signal from the controller, and
based on the received signal, generate a mask corresponding to each segment of the staged output data; and
a memory configured to store the staged output data using the generated masks.

### Claim 2 (depends on 1)

The accelerator of claim 1, wherein the data mask generator comprises a shift register.

### Claim 3 (depends on 1)

The accelerator of claim 1, wherein the pattern generation signal causes the data mask generator to execute a stride based on a type of the output data.

### Claim 4 (depends on 1)

The accelerator of claim 1, wherein the mask has a width equal to an output width of the systolic array.

### Claim 5 (depends on 1)

The accelerator of claim 1, wherein the memory is a dynamic random-access memory.

### Claim 6 (depends on 1)

The accelerator of claim 1, wherein each of the plurality of processing units is a multiply-accumulate unit.

### Claim 7 (depends on 1)

The accelerator of claim 1, wherein the data mask generator is further configured to generate the mask for 2n-1 consecutive writes.

### Claim 8 (depends on 1)

The accelerator of claim 1, wherein the memory is further configured to store merged output data, wherein the output data is merged with previous data output by the systolic array based on the mask associated with each segment of the staged output data.

### Claim 9 (depends on 8)

The accelerator of claim 8, wherein the systolic array is further configured to:
receive the merged output data, and
perform a subsequent calculation using the merged output data.

### Claim 10 (depends on 8)

The accelerator of claim 8, wherein the staged output data comprises a first data stream having a tail segment associated with a first mask and a second data stream having a head segment associated with a second mask.

### Claim 11 (depends on 10)

The accelerator of claim 10, wherein the merged output data is generated by combining the tail segment of the first data stream and the head segment of the second data stream based on the combination of the first and second masks.

### Claim 12

A method for systolic array-friendly data placement, comprising:
receiving, by a systolic array, staged input data,
performing an operation using the staged input data, thereby generating a first output matrix and a second output matrix;
generating a pattern generation signal;
generating a mask based on the pattern generation signal; and
storing merged output data comprising the first output matrix and the second output matrix.

### Claim 13 (depends on 12)

The method of claim 12, wherein the operation performed by the systolic array comprises a general matrix multiply (GeMM) operation.

### Claim 14 (depends on 12)

The method of claim 12, wherein the staged input data comprises one of: activation data or weight data.

### Claim 15 (depends on 12)

The method of claim 12, further comprising:
performing, by the systolic array, a second operation on the merged output data.

### Claim 16 (depends on 12)

The method of claim 12, wherein the staged input data is generated by shifting input data such that each column of input data is delayed by a clock cycle.

### Claim 17 (depends on 12)

The method of claim 12, wherein the pattern generation signal is based on a type of data output by the systolic array.

### Claim 18 (depends on 12)

The method of claim 12, further comprising:
deleting a portion of the one or more batches of data when the portion of the one or more batches of data is determined not to be used in a predetermined time period.

### Claim 19 (depends on 12)

The method of claim 12, further comprising:
generating a plurality of outputs by the plurality of processing strings in parallel.

### Claim 20

A non-transitory computer readable storage medium storing a set of instructions that are executable by at least one processor of a computing device to cause the computing device to perform a method for executing a convolutional neural network operation, the method comprising:
fetching one or more batches of data in a first memory;
regrouping the one or more batches of data into multiple work items, wherein a first work item partially overlaps one or more work items among the multiple work items; and
broadcasting the multiple work items to a processing array comprising a plurality of processing strings, wherein the first work item is transferred to two or more processing strings of the processing array.