Distributed machine learning hyperparameter optimization

One or more non-transitory computer readable media (NTCRM) comprising instructions for operating a training node in a distributed machine learning (ML) hyperparameter (HP) tuning system, the distributed ML HP tuning system comprising a manager node and a plurality pf training nodes, and wherein execution of the instructions by one or more processors of a computing system is to cause the computing system to:

• until an ML model convergence occurs,
• obtain, from a queue storing HP sets, an HP set for training a local copy of an ML model;
• train the local copy of the ML model using HPs of the HP set in parallel with one or more other training nodes of the distributed HP tuning system training other HPs of other HP sets;
• determine a performance value for the HP set based on performance of the training using the HPs;
• and
• send the performance value and the HP set to a manager node for generation of an additional HP set from the HP set based on an optimization algorithm.

A distributed hyperparameter (HP) tuning system, comprising:

• a manager node configured to:
• continuously estimate HP sets for a machine learning (ML) model using an optimization algorithm,
• store each of the estimated HP sets in a queue, and
• stop the estimation when a performance value of an HP set used to train the ML model converges;
• and
• a plurality of training nodes, wherein individual training nodes of the plurality of training nodes are configured to:
• obtain, from the queue, respective HP sets for training respective local instances of the ML model;
• train the respective local instances using respective HPs of the respective HP sets in parallel with other training nodes of the plurality of training nodes;
• determine respective performance values for the HP sets based on performance of the trained respective local instances;
• and
• send the respective performance values and the respective HP sets to the manager node for further estimation of HP sets.