Distributed machine learning method and system

A system for distributed machine learning model training in communication with a plurality of computation nodes, comprising:

• a processor and a memory in communication to the processor, the memory storing an instructions, wherein the processor, when executing the instructions, is configured to:
• receive a global parameter acquisition instruction from a first computation node;
• determine whether a difference between a sequence number of a current iteration round of the first computation node and a number of a current iteration round of a second computation node falls within a preset range of number of rounds;
• send a global parameter to the first computation node when the difference falls within the preset range of number of rounds;
• receive an updated parameter sent by the first computation node after the first computation node performs the current iteration round of the first computation node using the global parameter sent by the system;
• compute a computation delay parameter according to a timestamp for receiving the updated parameter by the system and a timestamp for receiving the global parameter by the first computation node;
• update the global parameter according to the computation delay parameter and the updated parameter from the first computation node to obtain an updated global parameter;
• and
• store the updated global parameter for next iteration rounds of training.

A method for distributed machine learning and model training using a first computation node among a plurality of computation nodes and having a processor and memory and in communication with a parameter server comprising a processor and memory, wherein each of the plurality of computation nodes is used to perform iterative rounds of training of a model, the method comprising:

• sending, by the first computation node, a global parameter acquisition instruction to the parameter server;
• receiving, by the first computation node, a global parameter sent by the parameter server upon determining by the parameter server that a difference between a sequence number of a current iteration round of the first computation node and a sequence number of a current iteration round of a second computation node falls within a preset range of number of rounds;
• performing, by the first computation node, the current iteration round of the first computation node using the global parameter sent by the parameter server;
• obtaining, by the first computation node, an updated parameter after the current iteration round of the first computation node;
• and
• sending, by the first computation node, the updated parameter to the parameter server.

A first computation node for distributed machine learning and model training among a plurality of computation nodes and in communication with a parameter server, comprising a processor and a memory in communication with the processor, the memory storing instructions, wherein the processor, when executing the instructions, is configured to:

• send a global parameter acquisition instruction to the parameter server;
• receive a global parameter sent by the parameter server upon determining by the parameter server that a difference between a sequence number of a current iteration round of the first computation node and a sequence number of a current iteration round of a second computation node falls within a preset range of number of rounds;
• perform the current iteration round of the first computation node using the global parameter sent by the parameter server;
• obtain an updated parameter after the current iteration round of the first computation node;
• and
• send the updated parameter to the parameter server.