Deep neural network model design enhanced by real-time proxy evaluation feedback
AssigneeIntel Corporation
InventorsDaniel J. Cummings, Sharath Nittur Sridhar
Life of this patent
Estimating the term…
Abstract
The present disclosure is related to artificial intelligence (AI), machine learning (ML), and Neural Architecture Search (NAS) technologies, and in particular, to Deep Neural Network (DNN) model engineering techniques that use proxy evaluation feedback. The DNN model engineering techniques discussed herein provide near real-time feedback on model performance via low-cost proxy scores without requiring continual training and/or validation cycles, iterations, epochs, etc. In conjunction with the proxy-based scoring, semi-supervised learning mechanisms are used to map proxy scores to various model performance metrics. Other embodiments may be described and/or claimed.
Claims25
One or more non-transitory computer readable media (NTCRM) comprising instructions for predicting performance metrics for machine learning (ML) models, wherein execution of the instructions by one or more processors of a computing system is to cause the computing system to:
- generate a set of ML models for respective users;
- for each ML model in the set of ML models, operate a proxy feedback engine to generate a proxy score for a corresponding ML model in the set of ML models, wherein the proxy score of the corresponding ML model is based on one or more predicted performance metrics for the corresponding ML model;
- train a subset of ML models from the set of ML models to obtain one or more actual performance metrics for corresponding ML models in the subset of ML models;
- operate a semi-supervised learning (SSL) mechanism to learn relationships between the one or more actual performance metrics and the corresponding ML models in the subset of ML models;
- and
- update the proxy feedback engine with the learned relationships for generating proxy scores for another set of ML models.
An apparatus for providing a machine learning model development environment (MDE), the apparatus comprising:
- interface circuitry to obtain machine learning (ML) inputs from a client device;
- and
- processor circuitry communicatively coupled with the interface circuitry, wherein the processor circuitry is, in response to receipt of an individual input of the ML inputs from the client device, to:
- obtain an ML model based on the individual input,
- obtain a proxy score for the ML model from a proxy feedback engine, wherein:
- the proxy score for the ML model is based on a set of predicted performance metrics for the ML model,
- at least one of the ML model and the proxy score is used by a semi-supervised learning (SSL) mechanism to learn relationships between actual performance metrics and trained ML models, and
- the learned relationships are used by the proxy feedback engine for generating proxy scores for other ML models, and
- provide the ML model and the proxy score to the client device via the interface circuitry.
An apparatus employed as a proxy feedback engine of a machine learning (ML) computing system, the apparatus comprising:
- interface circuitry to obtain ML model configurations (MLMCs) via an ML model development environment (MDE);
- and
- processor circuitry communicatively coupled with the interface circuitry, wherein the processor circuitry is, for an individual MLMC of the obtained MLMCs, to:
- select one or more proxy functions from a set of proxy functions based on information in the individual MLMC,
- determine a proxy score for an ML model corresponding to the individual MLMC using the selected one or more proxy functions, wherein the proxy score is based on a set of predicted performance metrics for the ML model,
- provide the determined proxy score to a model analysis engine via the interface circuitry, and
- obtain, from the model analysis engine, an updated mapping of actual ML performance metrics to ML models based at least in part on the determined proxy score for generating updated proxy scores.
Dependencies are inferred from the claim text.