Systems and methods of resource configuration optimization for machine learning workloads

A system comprising:

• at least one processor;
• and
• a memory storing instructions that, when executed by the at least one processor, cause the system to perform a method comprising:
• determining a plurality of computing resource configurations used to perform machine learning model training jobs, wherein a computing resource configuration comprises:
• a first tuple including numbers of worker nodes and parameter server nodes, and
• a second tuple including resource allocations for the worker nodes and parameter server nodes;
• executing at least one machine learning training job using a first computing resource configuration having a first set of values associated with the first tuple, wherein during the executing the machine learning training job:
• monitors resource usage of the worker nodes and parameter server nodes caused by a second set of values associated with the second tuple;
• and
• determines whether to adjust the second set of values;
• determining whether a stopping criterion is satisfied;
• and
• selecting one of the plurality of computing resource configurations.

A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to perform:

• determining a plurality of computing resource configurations used to perform machine learning model training jobs, wherein a computing resource configuration comprises:
• a first tuple including numbers of worker nodes and parameter server nodes, and
• a second tuple including resource allocations for the worker nodes and parameter server nodes;
• executing at least one machine learning training job using a first computing resource configuration having a first set of values associated with the first tuple, wherein during the executing the machine learning training job:
• monitors resource usage of the worker nodes and parameter server nodes caused by a second set of values associated with the second tuple;
• and
• determines whether to adjust the second set of values;
• determining whether a stopping criterion is satisfied;
• and
• selecting one of the plurality of computing resource configurations.