# US 12,182,686 B2 · Neural hardware accelerator for parallel and distributed tensor computations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,686 B2
- Document ID: US12182686B2
- Status: Granted
- Kind: B2
- Application number: 15/967,482
- Filed: 2018-04-30
- Published: 2019-10-31
- Granted: 2024-12-31
- Earliest claimed: 2018-04-30
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Dharmendra S. Modha
- Web page: https://patentgrep.com/patent/US12182686B2

## CPC

- G06N 3/063: using electronic means
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06F 17/153: {Multidimensional correlation or convolution}
- G06N 3/048: Activation functions
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Networks and encodings therefor are provided that are adapted to provide increased energy efficiency and speed for convolutional operations. In various embodiments, a neural network comprises a plurality of neural cores. Each of the plurality of neural cores comprises a memory. A network interconnects the plurality of neural cores. The memory of each of the plurality of neural cores comprises at least a portion of a weight tensor. The weight tensor comprising a plurality of weights. Each neural core is adapted to retrieve locally or receive a portion of an input image, apply the portion of the weight tensor thereto, and store locally or send a result therefrom via the network to other of the plurality of neural cores.

## Claims

### Claim 1

A system comprising:
a plurality of neural cores, each of the plurality of neural cores comprising at least one memory, wherein the plurality if neural cores is arranged in a two-dimensional array, and wherein each of the plurality of neural cores includes direct connections and relay connections with other of the plurality of neural cores; and
a network interconnecting the plurality of neural cores, wherein:
the at least one memory of each of the plurality of neural cores comprises at least a portion of a weight tensor,
the weight tensor comprising a plurality of filters,
each neural core is adapted to retrieve locally or receive a portion of an input data tensor, apply the portion of the weight tensor thereto, and store locally or send a result therefrom via the network to other of the plurality of neural cores,
the weight tensor is distributed among the plurality of neural cores according to physical dimensions of the two-dimensional array and based on a locality of the weight tensor and the direct connections of each of the plurality of neural cores,
wherein the distribution of the weight tensor is according to a ratio of a number of the direct connections to a number of the relay connections, wherein the ratio is maximized for the direct connections to the relay connections when transmitting the weight tensor between the plurality of neural cores, and the dimensions, shape, and partitioning of the weight tensor are configurable.

### Claim 2 (depends on 1)

The system of claim 1, wherein applying the portion of the weight tensor to the portion of the input data tensor comprises computing a vector-matrix multiplication followed by an activation function.

### Claim 3 (depends on 1)

The system of claim 1, wherein the network provides local connectivity among the plurality of neural cores.

### Claim 4 (depends on 1)

The system of claim 1, wherein the network provides connectivity among neighboring neural cores within the array.

### Claim 5 (depends on 1)

The system of claim 1, wherein the portion of the weight tensor at each of the plurality of neural cores comprises a block of the weight tensor.

### Claim 6 (depends on 1)

The system of claim 1, wherein the weight tensor is partitioned along one or more dimensions exhibiting locality.

### Claim 7 (depends on 6)

The system of claim 6, wherein the one or more dimensions comprise spatial dimensions.

### Claim 8 (depends on 6)

The system of claim 6, wherein the one or more dimensions comprise at least one feature dimension.

### Claim 9 (depends on 1)

The system of claim 1, wherein the weight tensor is sparse.

### Claim 10 (depends on 9)

The system of claim 9, wherein during said applying the portion of the weight tensor to the portion of the input data tensor, only non-zero values of the weight tensor are used for computation.

### Claim 11 (depends on 9)

The system of claim 9, wherein the memory stores only non-zero values of the weight tensor.

### Claim 12 (depends on 1)

The system of claim 1, wherein the weight tensor corresponds to a convolution filter.

### Claim 13 (depends on 12)

The system of claim 12, wherein the weight tensor is partitioned along one or more spatial dimension.

### Claim 14 (depends on 1)

The system of claim 1, wherein the result is an intermediate result, and wherein:
each neural core is further adapted to store locally or send the intermediate result via the network to other of the plurality of neural cores;
each neural core is further adapted to retrieve locally or receive the intermediate result and compute a final result therefrom.

### Claim 15 (depends on 1)

The system of claim 1, wherein the portion of the weight tensor at each of the plurality of neural cores comprises a block of the weight tensor, and wherein a subset of the plurality of neural cores include the same replicated block.

### Claim 16 (depends on 1)

The system of claim 1, wherein one of the plurality of neural cores includes the whole weight tensor.

### Claim 17 (depends on 16)

The system of claim 16, wherein the one neural core is adapted to distribute the portions of the weight tensor to each other of the plurality of neural cores.

### Claim 18 (depends on 1)

The system of claim 1, further comprising:
a central memory comprising the whole weight tensor, wherein
the system is adapted to distribute the portions of the weight tensor to each other of the plurality of neural cores from the central memory.

### Claim 19 (depends on 1)

The system of claim 1, wherein the portion of the weight tensor corresponds to a subset of the plurality of filters.

### Claim 20 (depends on 1)

The system of claim 1, wherein the weight tensor is compressed.

### Claim 21 (depends on 1)

The system of claim 1, wherein the weight tensor is encoded to exclude zero values.

### Claim 22 (depends on 1)

The system of claim 1, wherein each value of the weight tensor is −1, 0, or 1.

### Claim 23

A method comprising:
retrieving locally or receiving a portion of an input data tensor at a neural core of a plurality of neural cores, the neural core comprising a memory, the plurality of neural cores being arranged in a two-dimensional array, and the plurality of neural cores including direct connections and relay connections with other of the plurality of neural cores;
reading from the memory at least a portion of a weight tensor, the weight tensor comprising a plurality of filters, the weight tensor being distributed among the plurality or neural cores according to physical dimensions of the two-dimensional array and based on a locality of the weight tensor and the direct connections of each of the plurality of neural cores, and the dimensions, shape, and partitioning of the weight tensor being configurable,
wherein the distribution of the weight tensor is according to a ratio of a number of the direct connections to a number of the relay connections;
wherein the ratio is maximized for the direct connections to the relay connections when transmitting the weight tensor between the plurality of neural cores;
applying the portion of the weight tensor to the portion of the input data tensor to obtain a result;
storing locally or sending the result via a network to at least one other neural core.

### Claim 24 (depends on 23)

The method of claim 23, wherein the weight tensor is sparse.

### Claim 25 (depends on 23)

The method of claim 23, wherein the portion of the weight tensor corresponds to a subset of the plurality of filters.

### Claim 26 (depends on 23)

The method of claim 23, wherein the weight tensor is compressed.

### Claim 27 (depends on 23)

The method of claim 23, wherein the weight tensor is encoded to exclude zero values.

### Claim 28 (depends on 23)

The method of claim 23, wherein each value of the weight tensor is −1, 0, or 1.

### Claim 29

A system comprising:
a plurality of neural cores, each of the plurality of neural cores comprising a memory,
wherein the plurality of neural cores is arranged in a two-dimensional array, and wherein each of the plurality of neural cores includes direct connections and relay connections with other of the plurality of neural cores;
a network interconnecting the plurality of neural cores;
a computing node comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor of the computing node to cause the processor to perform a method comprising:
encoding a plurality of filters in a weight tensor; and
providing at least a portion of the weight tensor to each of the plurality of neural cores,
wherein the weight tensor is distributed among the plurality of neural cores according to physical dimensions of the two-dimensional array and based on a locality of the weight tensor and the direct connections of each of the plurality of neural cores, and the dimensions, shape, and partitioning of the weight tensor are configurable, wherein the
distribution of the weight tensor is according to a ratio of a number of the direct connections to a number of the relay connections,
wherein the ratio is maximized for the direct connections to the relay connections when transmitting the weight tensor between the plurality of neural cores, and
wherein each of the plurality of neural cores is adapted to:
store in its memory at least a portion of the weight tensor;
retrieve locally or receive a portion of an input data tensor;
apply the portion of the weight tensor to the portion of the input data tensor to obtain a result; and
store locally or send the result via a network to at least one other neural core.