# US 2019/0244105 A1 · Systolic neural network engine capable of backpropagation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244105 A1
- Document ID: US20190244105A1
- Status: Published application
- Kind: A1
- Application number: 15/981,664
- Filed: 2018-05-16
- Published: 2019-08-08
- Earliest claimed: 2018-02-08
- Assignees: Western Digital Technologies, Inc.
- Applicants: Western Digital Technologies, Inc.
- Inventors: Luiz M. Franca-Neto
- Web page: https://patentgrep.com/patent/US20190244105A1

## CPC

- G06N 3/063: using electronic means
- G06F 15/8046: {Systolic arrays}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 5/046: Forward inferencing; Production systems
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method of computer processing is disclosed comprising receiving a data packet at a processing node of a neural network, performing a calculation of the data packet at the processing node to create a processed data packet, attaching a tag to the processed data packet, transmitting the processed data packet from the processing node to a receiving node during a systolic pulse, receiving the processed data packet at the receiving node, performing a clockwise convolution on the processed data packet and a counter clockwise convolution on the processed data packet, performing an adding function and backpropagating results of the performed sigmoid function to each of the processing nodes that originally processed the data packet.

## Claims

### Claim 1

A device for performing training of a neural network via backward propagation, the device comprising:
a memory configured to store input values and corresponding expected output values;
at least one additional memory configured to store weights corresponding to connections between nodes of the neural network; and
a plurality of processing units that perform computations according to particular nodes of the neural network;
wherein the plurality of processing nodes are configured to perform computations for forward propagation of the input values through layers of the neural network to generate predicted output values by at least, for particular processing units of the plurality of processing units:
receiving input data;
processing the input data using one or more of the weights to generate an activation output; and
generating output data including the activation output and a tag identifying the particular processing unit; and
wherein the plurality of processing nodes are configured to perform computations for backward propagation of differences between the expected output values and corresponding ones of the predicted outputs at least partly based on the tags of the particular processing units, wherein the backward propagation updates the weights.

### Claim 2 (depends on 1)

The device of claim 1, wherein the device includes a systolic processor chip, and wherein the plurality of processing units comprise circuitry embedded in the systolic processor chip.

### Claim 3 (depends on 2)

The device of claim 2, wherein particular subsets of the plurality of processing units form particular arrangements of a systolic array, wherein particular arrangements of the systolic array are assigned to particular layers of the neural network.

### Claim 4 (depends on 3)

The device of claim 3, wherein a first arrangement of first processing units of the systolic array is configured to systolically pulse values output by the first processing units to a second arrangement of the systolic array during the forward propagation.

### Claim 5 (depends on 4)

The device of claim 4, wherein the first processing units are configured to accumulate partial derivatives based on values received from the second arrangement during the backward propagation.

### Claim 6 (depends on 2)

The device of claim 2, wherein the systolic processor chip is configured to:
systolically pulse data in a first direction through the plurality of processing units during the forward propagation, and
systolically pulse data in a second direction through the plurality of processing units during the backward propagation, wherein the second direction is opposite the first direction.

### Claim 7 (depends on 1)

The device of claim 1, wherein the at least one additional memory storing weights comprises a plurality of local memories within corresponding processing units.

### Claim 8 (depends on 1)

The device of claim 1, wherein the at least one additional memory storing weights is disposed external to the plurality of processing units, and wherein the plurality of processing units are configured to fetch identified ones of the weights from the at least one additional memory.

### Claim 9 (depends on 1)

The device of claim 1, wherein a particular processing unit of the plurality of processing nodes includes an input systolic element configured to receive the input data, processing circuitry configured to perform processing on the received input data to generate the activation output, and an output systolic element configured to systolically output the activation output.

### Claim 10 (depends on 1)

The device of claim 1, wherein the plurality of processing nodes are arranged in a plurality of arrangements, wherein each of the plurality of arrangements is configured to perform computations of a corresponding layer of the neural network.

### Claim 11 (depends on 10)

The device of claim 10, wherein a particular arrangement of the plurality of arrangements includes a first subset of the plurality of processing nodes, wherein, during the backward propagation, the first subset of plurality of processing nodes are configured to compute partial derivatives based at least partly on the tags, wherein a weighted sum of the partial derivatives is accumulated as the data is propagated backwards through the particular arrangement.

### Claim 12

A method for training of a neural network via an array of systolic processing units, the method comprising:
accessing input values and corresponding expected output values for the neural network;
computing, by the array of systolic processing units, a forward propagation of the input values through layers of the neural network to generate predicted output values, wherein performing the forward propagation includes:
computing particular activation output values using particular systolic processing units using at least some weights of the neural network; and
generating tagging information representing which of the systolic processing units computed which particular activation output value;
computing, by the array of systolic processing units, a backward propagation of differences between the expected output values and corresponding ones of the predicted output values, wherein computing the backward propagation is based at least partly on the tagging information; and
updating the weights based on the backward propagation.

### Claim 13 (depends on 12)

The method of claim 12, wherein the systolic processing units are arranged in arrangements, the method further comprising assigning particular arrangements to particular layers of the neural network.

### Claim 14 (depends on 13)

The method of claim 13, wherein computing the forward propagation further comprises systolically pulsing activation outputs of a first arrangement of the arrangements to a second arrangement of the arrangements.

### Claim 15 (depends on 14)

The method of claim 14, wherein computing the backward propagation further comprises systolically pulsing deviation outputs of the second arrangement to the first arrangement.

### Claim 16 (depends on 15)

The method of claim 15, wherein computing the backward propagation further comprises, by particular systolic processing units of the first arrangement, computing partial derivatives based at least partly on the tagging information.

### Claim 17 (depends on 16)

The method of claim 16, further comprising accumulating a weighted sum of the partial derivatives as data is propagated backwards through the first arrangement.

### Claim 18 (depends on 12)

The method of claim 12, wherein generating the tagging information comprises, by a first processing unit of the systolic processing units, tagging an activation value output from the first processing unit with an address of the first processing unit.

### Claim 19 (depends on 18)

The method of claim 18, wherein performing the forward propagation comprises, by a second processing unit of the systolic processing units, identifying one of the weights to use for processing the activation value based on the address of the first processing unit.

### Claim 20 (depends on 18)

The method of claim 18, wherein performing the backward propagation comprises updating the one of the weights based on the address of the first processing unit.
