Distributed weight update for backpropagation of a neural network

One or more processors, comprising:

• circuitry to:
• distribute, to a plurality of different workers, gradients associated with weights of one or more neural networks;
• cause each of the plurality of different workers to compute, using the gradients received, updates to a respective non-overlapping set of the weights of the one or more neural networks;
• and
• cause a complete set of updated weights of the one or more neural networks to be generated based, at least in part, on merging of the updates to the respective non-overlapping set of the weights of the one or more neural networks.

A system, comprising:

• one or more processors to:
• distribute, among a plurality of different workers, gradients of weights of one or more neural networks;
• cause individual ones of the plurality of different workers to compute, using the gradients, updates to a respective non-overlapping set of weights of the one or more neural networks;
• and
• cause a complete set of updated weights of the one or more neural networks to be generated based, at least in part, on merging of the updates to the respective non-overlapping set of the weights of the one or more neural networks.

A method, comprising:

• distributing, to a plurality of different workers, gradients associated with weights of one or more neural networks;
• causing each of the plurality of different workers to compute, using the gradients received, updates to a respective non-overlapping set of the weights of the one or more neural networks;
• and
• causing a complete set of updated weights of the one or more neural networks to be generated based, at least in part, on merging of the updates to the respective non-overlapping set of the weights of the one or more neural networks.

A hardware accelerator, comprising:

• circuitry to:
• receive gradients associated with weights of one or more neural networks;
• compute updates to a set of the weights of the one or more neural networks using the gradients, wherein the set of weights is non-overlapping with respect to other updates, to other sets of weights of the one or more neural network, to be computed by one or more other hardware accelerators;
• transmit the updates to one or more additional hardware accelerators;
• and
• generate a set of updated weights of the one or more neural networks based at least on updates received from the one or more other hardware accelerators.