Concurrent optimization of machine learning model performance

A system, comprising:

• one or more processors;
• and
• a memory having instructions stored thereon which, when executed by the one or more processors configured individually or in any combination, performs an operation for performing inferences using a machine learning model on a computing device, the operation comprising:
• receiving a request to perform inferences on a data set using a machine learning model configured to perform the inferences and performance metric targets for performance of the inferences;
• performing a first inference on the data set using the machine learning model to meet a latency specified for generation of the first inference from receipt of the request, wherein the first inference is performed using baseline operational parameters configured for execution of inference operations using the machine learning model, wherein the baseline operational parameters comprise predefined parameters for the machine learning model;
• while performing the first inference using the machine learning model, identifying operational parameters using an operational parameter optimizer model configured to identify the operational parameters resulting in inference performance approaching, meeting, or exceeding the performance metric targets based on the machine learning model and operational properties of a computing device;
• and
• applying the identified operational parameters to performance of subsequent inferences using the machine learning model.

A non-transitory computer-readable medium having instructions stored thereon which, when executed by one or more processors configured individually or in any combination, performs an operation for performing inferences using a machine learning model on a computing device, the operation comprising:

• receiving a request to perform inferences on a data set using a machine learning model configured to perform the inferences and performance metric targets for performance of the inferences;
• performing a first inference on the data set using the machine learning model to meet a latency specified for generation of the first inference from receipt of the request, wherein the first inference is performed using baseline operational parameters configured for execution of inference operations using the machine learning model, wherein the baseline operational parameters comprise predefined parameters for the machine learning model;
• while performing the first inference using the machine learning model, identifying operational parameters using an operational parameter optimizer model configured to identify the operational parameters resulting in inference performance approaching, meeting, or exceeding the performance metric targets based on the machine learning model and operational properties of a computing device;
• and
• applying the identified operational parameters to performance of subsequent inferences using the machine learning model.