Method and system for distributed training using synthetic gradients

A training node, comprising:

• a first processor coupled to a first memory;
• a second processor coupled to a second memory;
• and
• a synthetic gradient processing unit (SGPU) coupled to a third memory, the first processor and the second processor,
• wherein at least a portion of an electronic model is disposed in the first memory, the second memory, and the third memory,
• wherein the SGPU generates a synthetic gradient signal based on an error data signal from the first processor and the at least a portion of the electronic model, and
• wherein the synthetic gradient signal is configured to update the electronic model during a training operation for the electronic model.

A method, comprising:

• obtaining, by a distributed training controller, training data and an electronic model;
• determining, by the distributed training controller and based on a distribution algorithm, a resource distribution for updating the electronic model using a plurality of training nodes,
• wherein at least one training node among the plurality of training nodes comprises a synthetic gradient processing unit (SGPU), and
• wherein the electronic model is updated based on a synthetic gradient signal that is generated by the SGPU in response to an error data signal;
• generating, using the plurality of training nodes, the training data, and the resource distribution, a trained model based on the electronic model.