Back to results
US 11,775,867 B1Granted
System and methods for evaluating machine learning models
AssigneeSystem S.P.A.
InventorMehdi Jamei
Life of this patent
Estimating the term…
Abstract
A system and methods to provide an independent and unbiased service that comprehensively analyzes the performance of a predictive machine learning model and enables the performance characteristics of the model to be compared to other models and to relevant benchmarks.
Claims18
A method for evaluating a machine learning model, comprising:
- receiving a request to evaluate a machine learning model at a server operated by a provider of a machine learning model evaluation service;
- receiving the machine learning model and a set of testing data from s sender of the request, wherein the testing data is to be used to evaluate the model at the server;
- determining a set of one or more tests to perform to evaluate the machine learning model;
- converting one or more of the machine learning model and the set of testing data to a different format than the one they are in when provided by the sender of the request;
- enabling the set of tests to be performed and associated with the request, wherein the set of tests comprises one or more of generating model performance metrics, feature importance metrics, partial dependence plots, interaction metrics between different features, and descriptive statistics;
- performing each test of the set of tests on the machine learning model, wherein performing a test generates an evaluation metric;
- and
- providing each of the generated evaluation metrics to a sender of the request.
5 more elements
A system, comprising:
- an electronic processor or processors coupled to one or more non-transitory computer readable media;
- and the one or more non-transitory computer-readable media, wherein the non-transitory computer-readable media contains a set of computer-executable instructions, and when the electronic processor or processors are programmed with and execute the set of instructions, the system is configured to:
- receive a request to evaluate a trained machine learning model;
- receive the trained machine learning model and a set of testing data from a sender of the request wherein the testing data is to be used to evaluate the model:
- determine a set of one or more tests to perform to evaluate the machine learning model;
- convert one or more of the machine learning model and the set of testing data to a different format than the one they are in when provided by the sender of the request;
- enable the set of tests to be performed and associated with the request, wherein the set of tests comprises one or more of generating model performance metrics, feature importance metrics, partial dependence plots, interaction metrics between different features, and descriptive statistics;
- perform each test of the set of tests on the machine learning model, wherein performing a test generates an evaluation metric;
- and
- provide each of the generated evaluation metrics to the sender of the request.
7 more elements
A set of one or more non-transitory computer-readable media comprising a set of computer-executable instructions that when executed by one or more programmed electronic processors, cause the processors or a device or apparatus in which they are contained to:
- receive a request to evaluate a machine learning model:
- receive the machine learning model and a set of testing data to be used to evaluate the model;
- convert one or more of the machine learning model and the set of testing data to a different format than the one they are in when provided by the sender of the request;
- determine a set of one or more tests to perform to evaluate the machine learning model;
- enable the set of tests to be performed and associated with the request, wherein the set of tests comprises one or more of generating model performance metrics, feature importance metrics, partial dependence plots, interaction metrics between different features, and descriptive statistics;
- perform each test of the set of tests on the trained machine learning model, wherein performing a test generates an evaluation metric;
- and
- provide each of the generated evaluation metrics to a sender of the request.
5 more elements
Dependencies are inferred from the claim text.