Intelligent compute resource selection for machine learning training jobs

A computer-implemented method comprising:

• executing at least a portion of a desired machine learning (ML) training job a plurality of times using training data and a plurality of different resource configurations, the plurality of different resource configurations including a different type, a different amount, or both a different type and a different amount, of virtual machines;
• measuring, for each of the plurality of times the at least a portion of the desired ML training job is executed, a measured performance metric for each of the plurality of different resource configurations;
• generating, using a ML model trained on measured performance metrics from a plurality of other ML training jobs, and using the measured performance metrics from the desired ML training job as input to the ML model, and based at least in part on a desired performance characteristic, a recommended resource configuration for the desired ML training job;
• sending data indicating one or more recommended resource configurations for the desired ML training job to an electronic device of a user, the one or more recommended resource configurations including the recommended resource configuration;
• receiving a request to execute the desired ML training job from the electronic device of the user, wherein the request to execute the desired ML training job identifies the recommended resource configuration;
• and
• executing the desired ML training job using the recommended resource configuration.

A system comprising:

• a model training system implemented by a first one or more electronic devices;
• and
• a training configuration system implemented by a second one or more electronic devices, the training configuration system including second instructions that upon execution cause the training configuration system to:
• execute, via the model training system, at least a portion of a desired machine learning (ML) training job a plurality of times using training data and a plurality of different resource configurations, the plurality of different resource configurations including a different type, a different amount, or both a different type and a different amount, of virtual machines;
• measure, for each of the plurality of times the at least a portion of the desired ML training job is executed, a measured performance metric for each of the plurality of different resource configurations;
• generate, using a ML model trained on measured performance metrics from a plurality of other ML training jobs, and using the measured performance metrics from the desired ML training job as input to the ML model, and based at least in part on a desired performance characteristic, a recommended resource configuration for the desired ML training job;
• send data indicating one or more recommended resource configurations for the desired ML training job to an electronic device of a user, the one or more recommended resource configurations including the recommended resource configuration;
• receive a request to execute the desired ML training job from the electronic device of the user, wherein the request to execute the desired ML training job identifies the recommended resource configuration;
• and
• execute, via the model training system, the desired ML training job using the recommended resource configuration.