Distributed synchronous training architecture using stale weights

A distributed synchronous training system for training a neural network model, the system comprising:

• a plurality of worker machines, each worker machine of the plurality of worker machines comprising:
• a memory that stores instructions;
• and
• one or more processors in communication with the memory,
• wherein the one or more processors execute the instructions to:
• perform a forward computation of a training data set using a plurality of N layers of the neural network model, the forward computation starting at Layer 1 and proceeding through Layer N of the neural network model;
• perform a backward computation of the training data set, the backward computation starting at Layer N and proceeding through Layer 1 of the neural network model;
• and
• synchronize a plurality of gradients outputted by the neural network model during the backward computation, the synchronizing of the plurality of gradients performed with other worker machines of the plurality of worker machines and in parallel with the backward computation.

A computer-readable medium storing computer instructions for training a neural network model, wherein the instructions when executed by one or more processors of a worker machine of a plurality of worker machines in a deep-learning training architecture, cause the one or more processors to perform operations comprising:

• performing a forward computation of a training data set using a plurality of N layers of the neural network model, the forward computation starting at Layer 1 and proceeding through Layer N of the neural network model;
• performing a backward computation of the training data set, the backward computation starting at Layer N and proceeding through Layer 1 of the neural network model;
• and
• synchronizing, with other worker machines of the plurality of worker machines and in parallel with the backward computation, a plurality of gradients outputted by the neural network model during the backward computation.