Network-centric architecture and algorithms to accelerate distributed training of neural networks

A method of operating a distributed network of a first group of computing devices comprising computing devices circularly and sequentially coupled from a first computing device, to at least a second computing device, and to a final computing device coupled to the first computing device, and a second group of computing devices comprising computing devices circularly and sequentially coupled from a second first computing device to a second final computing device, wherein the second final computing device is also coupled to the second first computing device, the method comprising:

• sequentially aggregating, by respective ones of the first group of computing devices, gradient values from the first computing device to the final computing device, wherein the gradient values are data for training a neural network model and stored in each computing device of the first group of computing devices before the aggregating begins, and wherein the sequentially aggregating the gradient values comprises each computing device of the first group of computing devices, except the first computing device:
• combining locally stored gradient values with a partial aggregate of gradient values received from a previous device of the first group of computing devices;
• and
• storing the combined gradient values locally;
• wherein storing the aggregate gradient values in each computing device of the first group of computing devices results in a first aggregate of gradient values being buffered at the first computing device;
• transferring, by the first computing device, the first aggregate of gradient values to the second group of computing devices;
• and
• generating, by the second group of computing devices, from the first aggregate of gradient values, a second aggregate of gradient values to be used to train the neural network model.