Adaptive quantization for execution of machine learning models

A system, comprising:

• a processor;
• and
• a memory having instructions stored thereon which, when executed by the processor, performs an operation for adaptively executing machine learning models on a computing device, the operation comprising:
• receiving weight information for a model to be executed on a computing device;
• reducing the received weight information into quantized weight information having a reduced bit size relative to the received weight information;
• performing first inferences using the machine learning model and the received weight information;
• performing second inferences using the machine learning model and the quantized weight information;
• comparing results of the first and second inferences;
• determining that results of the second inferences are within a threshold performance level of results of the first inferences;
• and
• based on determining that results of the second inferences are within a threshold performance level of results of the first inferences, performing one or more subsequent inferences using the machine learning model and the quantized weight information.

A system for adaptively executing machine learning models on a computing device, comprising:

• means for receiving weight information for a machine learning model to be executed on a computing device;
• means for reducing the received weight information into quantized weight information having a reduced bit size relative to the received weight information;
• means for performing first inferences using the machine learning model and the received weight information;
• means for performing second inferences using the machine learning model and the quantized weight information;
• means for comparing results of the first and second inferences;
• means for determining that results of the second inferences are within a threshold performance level of results of the first inferences;
• and
• means for performing one or more subsequent inferences, based on determining that results of the second inferences are within a threshold performance level of results of the first inferences, using the machine learning model and the quantized weight information.

A computer-readable medium having instructions stored thereon which, when executed, performs an operation for adaptively executing machine learning models on a computing device, comprising:

• receiving weight information for a machine learning model to be executed on a computing device;
• reducing the received weight information into quantized weight information having a reduced bit size relative to the received weight information;
• performing first inferences using the machine learning model and the received weight information;
• performing second inferences using the machine learning model and the quantized weight information;
• comparing results of the first and second inferences;
• determining that results of the second inferences are within a threshold performance level of results of the first inferences;
• and
• based on determining that results of the second inferences are within a threshold performance level of results of the first inferences, performing one or more subsequent inferences using the machine learning model and the quantized weight information.