Systolic neural network engine capable of backpropagation

A device for performing training of a neural network via backward propagation, the device comprising:

• a memory configured to store input values and corresponding expected output values;
• at least one additional memory configured to store weights corresponding to connections between nodes of the neural network;
• and
• a plurality of processing units that perform computations according to particular nodes of the neural network;
• wherein the plurality of processing nodes are configured to perform computations for forward propagation of the input values through layers of the neural network to generate predicted output values by at least, for particular processing units of the plurality of processing units:
• receiving input data;
• processing the input data using one or more of the weights to generate an activation output;
• and
• generating output data including the activation output and a tag identifying the particular processing unit;
• and
• wherein the plurality of processing nodes are configured to perform computations for backward propagation of differences between the expected output values and corresponding ones of the predicted outputs at least partly based on the tags of the particular processing units, wherein the backward propagation updates the weights.