US 12,602,615 B2Granted

Evaluation of machine learning models using agreement scores

AssigneeDell Products L.P.

InventorsIam Palatnik de Sousa, Werner Spolidoro Freund, João Victor da Fonseca Pinto

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques are provided for evaluation of machine learning models using agreement scores. One method comprises obtaining two or more of: (i) a first set of quantitative features characterizing model parameters of a machine learning model; (ii) a second set of quantitative features characterizing a training process used to train the machine learning model; and (iii) a third set of quantitative features characterizing a training dataset used to train the machine learning model; generating a score based on an aggregation of at least portions of the two or more of the first set, the second set and the third set, wherein the score is based on an agreement of the machine learning with designated characteristics; and initiating an automated action based on the score. The automated action may comprise updating the machine learning model; generating a notification in connection with an audit; and/or selecting a machine learning model for deployment.

Claims20

Dependent claims

A method, comprising:

obtaining two or more of:
(i) a first set comprising one or more first quantitative feature values characterizing one or more model parameters of at least one machine learning model;
(ii) a second set comprising one or more second quantitative feature values characterizing a training process used to train the at least one machine learning model;
and (iii) a third set comprising one or more third quantitative feature values characterizing a training dataset used to train the at least one machine learning model;
generating a score based at least in part on an aggregation of at least portions of each of the two or more of:
the first set, the second set and the third set, wherein the score is based at least in part on an agreement of the at least one machine learning model with one or more designated characteristics;
initiating at least one automated action based at least in part on the score, wherein the at least one automated action comprises updating at least a portion of the at least one machine learning model based at least in part on the score;
and
initiating a deployment of the updated at least one machine learning model;
wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

7 more elements

An apparatus comprising:

at least one processing device comprising a processor coupled to a memory;
the at least one processing device being configured to implement the following steps:
obtaining two or more of:
(i) a first set comprising one or more first quantitative feature values characterizing one or more model parameters of at least one machine learning model;
(ii) a second set comprising one or more second quantitative feature values characterizing a training process used to train the at least one machine learning model;
and (iii) a third set comprising one or more third quantitative feature values characterizing a training dataset used to train the at least one machine learning model;
generating a score based at least in part on an aggregation of at least portions of each of the two or more of:
the first set, the second set and the third set, wherein the score is based at least in part on an agreement of the at least one machine learning model with one or more designated characteristics;
initiating at least one automated action based at least in part on the score, wherein the at least one automated action comprises updating at least a portion of the at least one machine learning model based at least in part on the score;
and
initiating a deployment of the updated at least one machine learning model.

8 more elements

A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps:

obtaining two or more of:
(i) a first set comprising one or more first quantitative feature values characterizing one or more model parameters of at least one machine learning model;
(ii) a second set comprising one or more second quantitative feature values characterizing a training process used to train the at least one machine learning model;
and (iii) a third set comprising one or more third quantitative feature values characterizing a training dataset used to train the at least one machine learning model;
generating a score based at least in part on an aggregation of at least portions of each of the two or more of:
the first set, the second set and the third set, wherein the score is based at least in part on an agreement of the at least one machine learning model with one or more designated characteristics;
initiating at least one automated action based at least in part on the score, wherein the at least one automated action comprises updating at least a portion of the at least one machine learning model based at least in part on the score;
and
initiating a deployment of the updated at least one machine learning model.

6 more elements

Dependencies are inferred from the claim text.