Method and device for training model in distributed system

A computing node in a distributed system, comprising:

• a first parameter server configured to store a sub-model of a model, wherein the sub-model is obtained by splitting the model, wherein the sub-model stored on the first parameter server is different from a sub-model stored on a second parameter server in another computing node in the distributed system wherein the distributed system comprises a master node and a plurality of slave nodes, and wherein each slave node comprises at least one parameter server and at least one parameter client;
• and
• a first parameter client configured to perform at least one round of training on the sub-model stored on the first parameter server to obtain a first training result of the sub-model,
• wherein the first parameter server is further configured to:
• receive the first training result and a second training result from a second parameter client in the other computing node;
• and
• update the first sub-model stored on the first parameter server based on the first training result and the second training result to obtain an updated sub-model.

A distributed system for training a model, comprising:

• a master node configured to split the model into a plurality of sub-models;
• and
• a plurality of slave nodes, wherein each slave node comprises at least one parameter server and at least one parameter client, wherein each parameter server stores a sub-model of the model, wherein the plurality of slave nodes comprises a first slave node and a second slave node, wherein a second parameter client in the second slave node is configured to:
• perform at least one round of training on a sub-model stored on a second parameter server in the second slave node to obtain a second training result, and
• send the second training result to a first parameter server in the first slave node,
• wherein the first parameter server in the first slave node is configured to:
• obtain a first training result by performing at least one round of training on a sub-model stored on the first parameter server in the first slave node;
• and
• update the sub-model stored on the first parameter server based on the first training result and the second training result to obtain an updated sub-model.