Benchmarking machine learning models via performance feedback

A computer-implemented method comprising:

• generating an execution plan for a benchmarking job of at least one training or inference task corresponding to a machine learning model based on an identified machine learning model, identified training data, and at least one objective for the benchmarking job;
• receiving execution statistics about the execution of the at least one training or inference task performed by the machine learning model as a part of the benchmarking job according to the execution plan;
• and
• updating the execution plan based at least in part on the received execution statistics of the at least one training or inference task.

A system comprising:

• execution resources to implement a benchmarking job according to an execution plan;
• and
• a benchmarking service implemented by a second one or more electronic devices, the benchmarking service including instructions that upon execution cause the benchmarking service to:
• generate an execution plan for a benchmarking job of at least one training or inference task corresponding to a machine learning model based on an identified machine learning model, identified training data, and at least one objective for the benchmarking job;
• receive execution statistics about the execution of the at least one training or inference task performed by the machine learning model as a part of the benchmarking job according to the execution plan;
• and
• update the execution plan based at least in part on the received execution statistics of the at least one training or inference task.