# US 12,182,691 B1 · Increasing performance of computational array accelerators

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,691 B1
- Document ID: US12182691B1
- Status: Granted
- Kind: B1
- Application number: 17/249,900
- Filed: 2021-03-17
- Granted: 2024-12-31
- Earliest claimed: 2021-03-17
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Sundeep Amirineni; Akshay Balasubramanian; Joshua Wayne Bowman; Ron Diamant; Paul Gilbert Meyer; Thomas Elmer
- Web page: https://patentgrep.com/patent/US12182691B1

## CPC

- G06N 3/063: using electronic means
- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 9/3001: {Arithmetic instructions}
- G06N 3/045: Combinations of networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

To improve performance of a computational array, the architecture of the array can be modified to allow the processing engines of a column to operate in parallel and the clock frequency of the array to be increased. The processing engines of each column of the array can be grouped into a series of row groups. The processing engines of each row group can be loaded with input values, and computations on the input values can be carried out in parallel to generate the column output. One or more flip-flop stages can be inserted into the computational logic of each of the processing engines. The computational logic can then be distributed across the flip-flop stages to reduce the propagation delay between flip-flop stages of the processing engine, hence allowing the clock frequency of the array to be increased.

## Claims

### Claim 1

A neural network processor comprising:
an array of processing engines having a plurality of columns each configured to generate a column sum output, wherein each of the columns includes:
a plurality of row groups each having a plurality of non-adjacent processing engines along the column, wherein each processing engine of the plurality of non-adjacent processing engines in a row group includes multiplication logic and addition logic distributed across one or more flip-flop stages, each processing engine configured to perform operations including:
receiving a feature map input value for computing the column sum output;
multiplying the feature map input value with a weight value to generate a multiplication result;
adding the multiplication result to a partial sum input to generate a partial sum output over a plurality of clock cycles; and
outputting the partial sum output from the processing engine to a next row group; and
a column adder to add partial sum outputs from a last row group to generate the column sum output.

### Claim 2 (depends on 1)

The neural network processor of claim 1, wherein the array is configured to be clocked with a clock signal having a clock period being equal to or greater than a maximum propagation delay between flip-flop stages of the addition logic.

### Claim 3 (depends on 1)

The neural network processor of claim 1, wherein the partial sum output is generated over at least three clock cycles.

### Claim 4 (depends on 1)

The neural network processor of claim 1, wherein the addition logic is distributed across the one or more flip-flop stages by a synthesis tool.

### Claim 5

An integrated circuit device comprising:
an array of processing engines having a plurality of columns each configured to generate a column output, wherein each column in the plurality of columns includes:
a plurality of row groups each having a plurality of non-adjacent processing engines along the column, wherein each processing engine of the plurality of non-adjacent processing engines in a row group includes a computational circuit and is configured to perform operations including:
receiving an input value for computing the column output;
performing a computation on the input value over a plurality of clock cycles using the computational circuit to derive an intermediate value; and
outputting the intermediate value from the processing engine to a next row group; and
a column accumulator to accumulate intermediate values outputted from a last row group to generate the column output.

### Claim 6 (depends on 5)

The integrated circuit device of claim 5, wherein the computation is a multiply-and-accumulate computation.

### Claim 7 (depends on 6)

The integrated circuit device of claim 6, wherein the column accumulator is an adder circuit.

### Claim 8 (depends on 6)

The integrated circuit device of claim 6, wherein the multiply-and-accumulate computation is performed over at least three clock cycles.

### Claim 9 (depends on 6)

The integrated circuit device of claim 6, wherein the input value is a feature map input value, the intermediate value is a partial sum output, and the multiply-and-accumulate computation is performed by multiplying the feature map input value with a weight value to derive a multiplication result, and by adding the multiplication result to a partial sum input to generate the partial sum output.

### Claim 10 (depends on 9)

The integrated circuit device of claim 9, wherein each processing engine of the plurality of non-adjacent processing engines includes a feature map input buffer, a partial sum input buffer, and a weight input buffer coupled to a weight register.

### Claim 11 (depends on 10)

The integrated circuit device of claim 10, wherein each processing engine of the plurality of non-adjacent processing engines includes a multiplier circuit coupled to an output of the feature map input buffer and an output of the weight register.

### Claim 12 (depends on 11)

The integrated circuit device of claim 11, wherein each processing engine of the plurality of non-adjacent processing engines includes an adder circuit coupled to an output of the partial sum input buffer and an output of the multiplier circuit.

### Claim 13 (depends on 5)

The integrated circuit device of claim 5, wherein the computational circuit includes computational logic distributed across multiple flip-flop stages.

### Claim 14 (depends on 13)

The integrated circuit device of claim 13, wherein the computational logic includes addition logic that is distributed across the multiple flip-flop stages.

### Claim 15 (depends on 13)

The integrated circuit device of claim 13, wherein the array of processing engines is configured to be clocked with a clock signal having a clock period being equal to or greater than a maximum propagation delay between the flip-flop stages of the computational circuit.

### Claim 16 (depends on 5)

The integrated circuit device of claim 5, wherein the plurality of non-adjacent processing engines of a row group are configured to receive their input values belonging to a vector of an input matrix in parallel.

### Claim 17 (depends on 5)

The integrated circuit device of claim 5, wherein each processing engine of a row group is configured to receive sequential input values on non-adjacent clock cycles.

### Claim 18 (depends on 5)

The integrated circuit device of claim 5, wherein each processing engine of a row group is configured to output sequential intermediate values to a next row group on non-adjacent clock cycles.

### Claim 19

A method for computing a column output in a column of processing engines of an array, wherein the column has a series of row groups that each include a plurality of non-adjacent processing engines along the column, the method comprising:
sequentially performing, by each row group in the series of row groups of the column:
loading, by each processing engine of the row group, an input value for computing the column output;
performing, by each processing engine of the row group, a computation on the input value to derive an intermediate value over a plurality of clock cycles; and
outputting, by each processing engine of the row group, the intermediate value to a next row group; and
accumulating intermediate values from a last row group to generate the column output.

### Claim 20 (depends on 19)

The method of claim 19, wherein sequential input values being loaded into a processing engine are separated by a plurality of clock cycles.