Dynamic distributed training of machine learning models

A computer program product comprising one or more computer readable storage media and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by one or more processors to cause the one or more processors to perform a method comprising:

• updating a machine learning model by synchronously applying, to the machine learning model, a first set of training results received at a sub-system from a set of trainers having respective training datasets, as synchronous training;
• receiving, at the sub-system, from one or more trainers of the set of trainers, a first set of metrics pertaining to at least some of the training results of the first set of training results;
• based, at least in part, on the first set of metrics, determining on a configuration computer communicating with the sub-system to subsequently update the machine learning model via asynchronous application of subsequent training results received from respective trainers of the set of trainers;
• in response to the first set of metrics indicating that an amount of time taken by a first trainer of the set of trainers to produce respective training results differs from an amount of time taken by one or more other trainers to produce respective training results by at least a threshold amount of time, switching, using the configuration computer, to an asynchronous training mode, the switching to the asynchronous training mode being based on the first trainer being a slow trainer;
• and
• switching back to the synchronous training, using the configuration computer, when the slow trainer is no longer the slow trainer when all the one or more trainers of the set of trainers report similar training times within a threshold distance.

A computer system comprising:

• one or more processors;
• and
• one or more computer readable storage media;
• wherein:
• the one are more processors are structured, located, connected and/or programmed to execute program instructions collectively stored on the one or more computer readable storage media;
• and
• the program instructions, when executed by the one or more processors, cause the one or more processors to perform a method comprising:
• updating a machine learning model by synchronously applying, to the machine learning model, a first set of training results received at a sub-system from a set of trainers having respective training datasets, as synchronous training;
• receiving at the sub-system, from one or more trainers of the set of trainers, a first set of metrics pertaining to at least some of the training results of the first set of training results;
• based, at least in part, on the first set of metrics, determining on a configuration computer communicating with the sub-system, to subsequently update the machine learning model via asynchronous application of subsequent training results received from respective trainers of the set of trainers;
• in response to the first set of metrics indicating that an amount of time taken by a first trainer of the set of trainers to produce respective training results differs from an amount of time taken by one or more other trainers to produce respective training results by at least a threshold amount of time, switching, using the configuration computer, to an asynchronous training mode, the switching to the asynchronous training mode being based on the first trainer being a slow trainer;
• and
• switching back to the synchronous training, using the configuration computer, when the slow trainer is no longer the slow trainer when all the one or more trainers of the set of trainers report similar training times within a threshold distance.