Parallelizing moment-based optimizations with blockwise model-update filtering

An electronic device, comprising:

• a processing unit;
• a memory coupled to the processing unit and storing instructions thereon, the instructions, when executed by the processing unit, performing acts comprising:
• providing, by a master node, a global model parameter and a global moment parameter to a plurality of worker nodes for a training cycle;
• receiving, from the plurality of worker nodes, a plurality of local model parameters and a plurality of local moment parameters, the plurality of local model parameters and the plurality of local moment parameters being generated by respective ones of the plurality of worker nodes performing moment-based optimizations in parallel for the training cycle based on the global model parameter and the global moment parameter;
• aggregating the plurality of local model parameters to obtain an aggregated model parameter and aggregating the plurality of local moment parameters to obtain an aggregated moment parameter;
• generating model update information for the training cycle based on the aggregated model parameter and historical model update information for a preceding training cycle;
• updating the global model parameter based on the model update information for the training cycle to obtain an updated global model parameter;
• updating the global moment parameter based on the aggregated moment parameter to obtain an updated global moment parameter compatible with the updated global model parameter;
• and
• providing the updated global model parameter and the updated global moment parameter to the plurality of worker nodes for performing moment-based optimizations in parallel for a succeeding training cycle.

A computer program product comprising executable instructions, the executable instructions, when executed on a device, cause the device to perform acts comprising:

• providing, by a master node, a global model parameter and a global moment parameter to a plurality of worker nodes for a training cycle;
• receiving, from the plurality of worker nodes, a plurality of local model parameters and a plurality of local moment parameters, the plurality of local model parameters and the plurality of local moment parameters being generated by respective ones of the plurality of worker nodes performing moment-based optimizations in parallel for the training cycle based on the global model parameter and the global moment parameter;
• aggregating the plurality of local model parameters to obtain an aggregated model parameter and aggregating the plurality of local moment parameters to obtain an aggregated moment parameter;
• generating model update information for the training cycle based on the aggregated model parameter and historical model update information for a preceding training cycle;
• updating the global model parameter based on the model update information for the training cycle to obtain an updated global model parameter;
• updating the global moment parameter based on the aggregated moment parameter to obtain an updated global moment parameter compatible with the updated global model parameter;
• and
• providing the updated global model parameter and the updated global moment parameter to the plurality of worker nodes for performing moment-based optimizations in parallel for a succeeding training cycle.