US 2019/0156247 A1Published application

Dynamic accuracy-based deployment and monitoring of machine learning models in provider networks

AssigneeAmazon Technologies, Inc.

InventorsThomas Albert FAULHABER, JR., Edo LIBERTY, Stefano STEFANI, Zohar KARNIN, Craig WILEY, Steven Andrew LOEPPKY, Swaminathan SIVASUBRAMANIAN, Alexander Johannes SMOLA, Taylor GOODHART

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Techniques for dynamic accuracy-based experimentation and deployment of machine learning (ML) models are described. Inference traffic flowing to ML models and the accuracy of the models is analyzed and used to ensure that better performing models are executed more often via model selection. A predictive component can evaluate which model is more likely to be accurate for certain input data elements. Ensemble techniques can combine inference results of multiple ML models to aim to achieve a better overall result than any individual model could on its own.

Claims20

Dependent claims

A method comprising:

obtaining a first plurality of inference results generated by a first machine learning (ML) model and a second plurality of inference results generated by a second machine learning (ML) model, wherein the first ML model and the second ML model are part a group of ML models associated with one or more users and that generate a common type of inference;
determining, based at least in part on the first plurality of inference results and the second plurality of inference results, a first accuracy score corresponding to the first ML model and a second accuracy score corresponding to the second ML model, wherein each accuracy score indicates an amount of correctness of the inferences generated by the corresponding ML model, wherein the first accuracy score indicates that the first ML model is of higher quality than the second ML model;
updating a model selector to cause the model selector to select between the first ML model and the second ML model for generating inferences for inference requests according to an updated distribution, the updated distribution indicating that a comparatively larger amount of inference requests are to be provided to the first ML model than as indicated by a previous distribution;
and
providing, by the model selector, a plurality of inference requests to the first ML model and the second ML model according to the updated distribution.

2 more elements

A system comprising:

a dynamic router implemented by a first one or more electronic devices, the dynamic router including first instructions that upon execution cause the dynamic router to implement a model selector to select one or more of a plurality of machine learning (ML) models according to a distribution to perform inferences for inference requests, and cause the inference requests to be provided to the selected ML models;
and
an analytics engine implemented by a second one or more electronic devices, the analytics engine including second instructions that upon execution cause the analytics engine to:
obtain a plurality of inference results generated by the plurality of ML models of a group that generate a common type of inference;
determine, based at least in part on the plurality of inference results, a plurality of accuracy scores corresponding to the plurality of ML models;
and
cause a model selector of the dynamic router to be updated, based on the plurality of accuracy scores, to use an updated distribution to select ones of the plurality of ML models to generate inferences for inference requests.

4 more elements

Dependencies are inferred from the claim text.