# US 12,008,469 B1 · Acceleration of neural networks with stacks of convolutional layers

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,469 B1
- Document ID: US12008469B1
- Status: Granted
- Kind: B1
- Application number: 17/009,517
- Filed: 2020-09-01
- Granted: 2024-06-11
- Earliest claimed: 2020-09-01
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Thiam Khean Hah; Randy Renfu Huang; Richard John Heaton; Ron Diamant; Vignesh Vivekraja
- Web page: https://patentgrep.com/patent/US12008469B1

## CPC

- G06N 3/08: Learning methods
- G06F 13/20: for access to input/output bus
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/063: using electronic means
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A single neural network model can be used by each computing engine (CE) in a neural network processor to perform convolution operations in parallel for one or more stacks of convolutional layers. An input feature map can be divided into N chunks to be processed by N CEs, respectively. Each CE can process a last portion of a respective chunk to generate respective shared states to be used by a subsequent CE. A first CE uses pre-computed states to generate a first portion of an output feature map, while other CEs use shared states computed by a preceding CE to generate respective portions of the output feature map.

## Claims

### Claim 1

A computer-implemented method to accelerate a dilated convolution neural network (CNN), the method comprising:
receiving a neural network model that includes convolution operations for generating an output dataset from an input dataset using N computing engines (CEs) executing in parallel, wherein the N CEs are part of a neural network processor, and wherein the input dataset is to be transferred to the neural network processor from an external memory and comprises N chunks of equal size, each chunk to be processed by a respective CE to generate a respective portion of the output dataset;
generating an instruction for obtaining pre-computed states to be used by a first CE from the N CEs; and
generating, based on the neural network model, a same set of instructions to be executed by each CE in parallel for:
processing a last portion of a respective chunk to compute respective shared states;
processing a middle portion of the respective chunk to generate a first sub-portion of the respective portion of the output dataset while receiving additional shared states for processing, wherein:
the additional shared states correspond, in instances where a first portion of the respective chunk overlaps with a last portion of a respective chunk for a preceding CE, to the respective shared states computed by the preceding CE and are received through transfer between a local memory of the CE and a local memory of the preceding CE,
the respective shared states computed by the preceding CE are received as a substitute for intermediate states that would otherwise have been computed by the CE using the first portion of the respective chunk, and
the first CE is able to execute the same set of instructions due to receiving the pre-computed states as the additional shared states; and
generating a second sub-portion of the respective portion of the output dataset using the received additional shared states.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the dilated CNN includes stacks of convolutional layers comprising a first stack and a second stack, the first stack comprising, from bottom to top, a first input layer, a first set of hidden layers, and a first output layer, the second stack comprising, from the bottom to the top, a second input layer, a second set of hidden layers, and a second output layer, and wherein the input dataset is processed by the first stack to generate the output dataset, and the output dataset is used as the input dataset for the second stack.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the same set of instructions are executed by each CE in parallel for each stack.

### Claim 4 (depends on 2)

The computer-implemented method of claim 2, wherein the pre-computed states include first pre-computed states and second pre-computed states, wherein the first CE receives the first pre-computed states as shared states for the first stack and the second pre-computed states as shared states for the second stack.

### Claim 5

A computer-implemented method comprising:
receiving a neural network model for a neural network, the neural network being configured to generate an output dataset from an input dataset using convolution operations performed by at least a first computing engine (CE) and a second CE, wherein the first CE and the second CE are part of a neural network processor, and wherein the input dataset is to be transferred to the neural network processor from an external memory and comprises a first chunk and a second chunk; and
generating, based on the neural network model, instructions for executing by the first CE and the second CE in parallel to:
process a last portion of a respective chunk to compute respective shared states;
process a middle portion of the respective chunk to generate a first sub-portion of a respective portion of the output dataset;
receive, while processing the middle portion of the respective chunk, additional shared states for processing, wherein:
the first chunk is the respective chunk for the first CE,
the second chunk is the respective chunk for the second CE,
the last portion of the first chunk overlaps with a first portion of the second chunk,
the additional shared states received by the second CE are the respective shared states computed by the first CE and are obtained from a local memory of the first CE,
the second CE receives the respective shared states computed by the first CE as a substitute for intermediate states that would otherwise have been computed by the second CE using the first portion of the second chunk, and
the first CE receives pre-computed states as the additional shared states; and
generate a second sub-portion of the respective portion of the output dataset using the received additional shared states.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, further comprising:
generating an instruction to obtain the pre-computed states to be used by the first CE.

### Claim 7 (depends on 5)

The computer-implemented method of claim 5, wherein the pre-computed states have known or static values.

### Claim 8 (depends on 5)

The computer-implemented method of claim 5, further comprising generating an additional instruction for executing by the first CE and the second CE in parallel to:
store the respective shared states in a respective local memory while processing the last portion of the respective chunk.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein:
the input dataset is a first input dataset in an input feature map for making an interference,
pre-computed states received by the first CE in connection with processing the first input dataset correspond to static or random values,
the input feature map further comprises a second input dataset having a respective first chunk and second chunk, and
pre-computed states received by the first CE in connection with processing the second input dataset correspond to shared states computed by, and stored in a local memory of, the second CE or another CE in the neural network processor during processing of the first input dataset.

### Claim 10 (depends on 5)

The computer-implemented method of claim 5, the method further comprising generating an additional instruction for executing by the first CE and the second CE in parallel to:
generate a third sub-portion of the respective portion of the output dataset using the computed respective shared states.

### Claim 11 (depends on 5)

The computer-implemented method of claim 5, wherein the second CE receives the respective shared states computed by the first CE through an internal bus of the neural network processor.

### Claim 12 (depends on 5)

The computer-implemented method of claim 5, wherein the neural network is a dilated convolutional neural network (CNN) comprising stacks of dilated convolutional layers, each stack comprising, from bottom to top, a respective input layer followed by a respective set of hidden layers and a respective output layer.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein the convolution operations are performed by each layer at a respective dilation rate for each stack that increases from the bottom to the top for each of the hidden layers and the output layer.

### Claim 14 (depends on 5)

The computer-implemented method of claim 5, wherein the first CE comprises a first systolic array including a first set of processing elements arranged in rows and columns to perform the convolution operations, and the second CE comprises a second systolic array including a second set of processing elements arranged in rows and columns to perform the convolution operations.

### Claim 15 (depends on 5)

The computer-implemented method of claim 5, wherein the first CE and the second CE are a first processor core and a second processor core of the neural network processor.

### Claim 16

A non-transitory computer readable medium having stored therein instructions that, when executed by one or more processors cause the one or more processors to execute a compiler, the compiler performing operations including:
receiving a neural network model for a neural network, the neural network being configured to generate an output dataset from an input dataset using convolution operations performed by at least a first computing engine (CE) and a second CE, wherein the first CE and the second CE are part of a neural network processor, and wherein the input dataset is to be transferred to the neural network processor from an external memory and comprises a first chunk and a second chunk; and
generating, based on the neural network model, instructions for executing by the first CE and the second CE in parallel to:
process a last portion of a respective chunk to compute respective shared states;
process a middle portion of the respective chunk to generate a first sub-portion of a respective portion of the output dataset;
receive, while processing the middle portion of the respective chunk, additional shared states for processing, wherein:
the first chunk is the respective chunk for the first CE,
the second chunk is the respective chunk for the second CE,
the last portion of the first chunk overlaps with a first portion of the second chunk,
the additional shared states received by the second CE are the respective shared states computed by the first CE and are obtained from a local memory of the first CE,
the second CE receives the respective shared states computed by the first CE as a substitute for intermediate states that would otherwise have been computed by the second CE using the first portion of the second chunk, and
the first CE receives pre-computed states as the additional shared states; and
generate a second sub-portion of the respective portion of the output dataset using the received additional shared states.

### Claim 17 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the compiler is executed on a first computer system, and wherein the neural network processor is an acceleration engine on a second computer system.

### Claim 18 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the first CE and the second CE are integrated circuits that include an on-chip memory for storing shared states, the on-chip memory of the first CE being the local memory that stores the additional shared states received by the second CE.

### Claim 19 (depends on 16)

The non-transitory computer readable medium of claim 16, the operations further including generating an instruction to obtain the pre-computed states to be used by the first CE.

### Claim 20

A method comprising:
receiving, by a first computing engine (CE) and a second CE in a neural network processor, an input dataset divided into chunks comprising a first chunk and a second chunk, each chunk to be processed by a respective CE to generate a respective portion of an output dataset using convolution operations, wherein:
the input dataset is transferred to the neural network processor from an external memory,
the first chunk and the second chunk each include a first portion, a middle portion, and a last portion,
the first chunk is the respective chunk for the first CE,
the second chunk is the respective chunk for the second CE, and
the last portion of the first chunk overlaps with the first portion of the second chunk;
processing, by each of the first CE and the second CE, the last portion of the respective chunk to compute respective shared states;
processing, by each of the first CE and the second CE, the middle portion of the respective chunk to generate a first sub-portion of the respective portion of the output dataset;
receiving, by each of the first CE and the second CE and while processing the middle portion of the respective chunk, additional shared states for processing, wherein:
the additional shared states received by the second CE are the respective shared states computed by the first CE and are obtained from a local memory of the first CE,
the second CE receives the respective shared states computed by the first CE as a substitute for intermediate states that would otherwise have been computed by the second CE using the first portion of the second chunk, and
the additional shared states received by the first CE are pre-computed states; and
generating, by each of the first CE and the second CE, a second sub-portion of the respective portion of the output dataset using the received additional shared states.