Elastic training of machine learning models via re-partitioning based on feedback from the training algorithm

A computerized system having an architecture adapted for assigning computing tasks to multiple workers of the system, the computerized system including at least one hardware processor, wherein the computerized system stores a computerized method of parallel training of a machine learning model, whereby the system is configured to:

• access training data;
• start a parallel training of the machine learning model based on the accessed training data, the parallel training distributed through a first number K of workers of the system, K>1;
• and
• in response to detecting a change in a temporal evolution of a quantity indicative of a convergence rate of the parallel training, the change reflecting a deterioration of the convergence rate and providing an indicator for deciding whether to scale-in the training for improving the convergence rate in training the machine learning model, scale-in the parallel training of the machine learning model, so as for the parallel training to be subsequently distributed through a second number K′ of workers, where K>K′≥1, K′ being determined to improve the convergence rate of the parallel training of the machine learning model, wherein K′ further changes, wherein successive scale-in operations are repeated wherein number of workers in the parallel training is elastic and becomes less throughout the parallel training, wherein the number of workers is dynamically selected repeatedly during the parallel training to bring the convergence rate to maximal at points in time during the parallel training of the machine learning model,
• wherein the quantity is a duality-gap measuring a distance between a primal formulation of a training objective for the training and a dual formulation of this training objective,
• wherein the change is detected by comparing at least two slopes of the temporal evolution of the duality-gap, the at least two slopes including a short-term slope and a long-term slope,
• wherein the long-term slope is indicative of the convergence of the duality-gap over a period of time extending since a last scale-in event operated at the computerized system during the parallel training,
• wherein the short-term slope is indicative of the convergence of the duality-gap over a period of time extending over a finite number N of one or more most recent training epochs of the parallel training, N≥1.

A computer program product for parallel training of a machine learning model on a computerized system, whose computing tasks can be assigned to multiple workers of the system, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable on the computerized system to cause the latter to:

• access training data;
• start a parallel training of the machine learning model based on the accessed training data, the parallel training distributed through a first number K of workers of the system, K>1;
• and
• in response to detecting a change in a temporal evolution of a quantity indicative of a convergence rate of the parallel training, the change reflecting a deterioration of the convergence rate and providing an indicator for deciding whether to scale-in the training for improving the convergence rate in training the machine learning model, scale-in the parallel training of the machine learning model, so as for the parallel training to be subsequently distributed through a second number K′ of workers, where K>K′≥1, K′ being determined to improve the convergence rate of the parallel training of the machine learning model, wherein K′ further changes, wherein successive scale-in operations are repeated wherein number of workers in the parallel training is elastic and becomes less throughout the parallel training, wherein the number of workers is dynamically selected repeatedly during the parallel training to bring the convergence rate to maximal at points in time during the parallel training of the machine learning model,
• wherein the quantity is a duality-gap measuring a distance between a primal formulation of a training objective for the training and a dual formulation of this training objective,
• wherein the change is detected by comparing at least two slopes of the temporal evolution of the duality-gap, the at least two slopes including a short-term slope and a long-term slope,
• wherein the long-term slope is indicative of the convergence of the duality-gap over a period of time extending since a last scale-in event operated at the computerized system during the parallel training,
• wherein the short-term slope is indicative of the convergence of the duality-gap over a period of time extending over a finite number N of one or more most recent training epochs of the parallel training, N≥1.