US 10,380,498 B1Granted

Platform services to enable one-click execution of the end-to-end sequence of modeling steps

AssigneeAmazon Technologies, Inc.

InventorsVineet Shashikant Chaoji, Aswin Natarajan, Seyit Ismail Parsa, Rajeev Ramnarain Rastogi

Life of this patent

Estimating the term…

Loading patent…

Abstract

This disclosure is directed to the automated generation of Machine Learning (ML) models. The system receives a user directive containing one or more requirements for building the ML model. The system further identifies common requirements between the user directive and one or more prior user directives and associates characteristics of the prior user directive, or model generated therefrom, with the user directive. The system further associates performance values generated by continuous monitoring of deployed ML models to individual characteristics of the user directive used to generate each of the deployed ML models. The system continuously improves model generation efficiency, model performance, and first run performance of individual ML models by learning from the improvements made to one or more prior ML models having similar characteristics.

Claims20

Dependent claims

A method to automatically generate machine learning (ML) models, the method comprising:

receiving a first user directive including one or more first model requirements;
identifying a prior user directive associated with one or more prior model requirements and a prior model, the one or more prior model requirements and the one or more first model requirements including at least one common requirement, wherein the at least one common requirement includes at least one of a common target feature or a common raw data source, the common target feature including an instruction predict a percentage of customers likely to sign up for a subscription service and the common raw data source including customer behavior data, and wherein identifying the prior user directive comprises:
comparing at least one first model requirement of the one or more first model requirements with at least one prior model requirement of the one or more prior model requirements,
determining, based at least in part on the comparing of the at least one first model requirement and the at least one prior model requirement, a similarity score associated with the prior user directive, and
selecting the prior user directive from a plurality of prior user directives based at least in part on the similarity score;
selecting, based at least in part on the prior user directive, one or more features used to build the prior model;
training, based at least in part on the prior user directive, a first model to determine one or more first model parameter values;
determining a performance value of the first model;
monitoring, via an at least partially automated process, the performance value to determine a distribution of the performance value over a period of time;
and
associating, based at least in part on the monitoring of the performance value, the performance value of the first model with at least one of the one or more features.

8 more elements

A system comprising:

one or more processors;
memory accessible by the one or more processors;
and
one or more computer-executable instructions stored in the memory and executable on the one or more processors to perform operations comprising:
receiving a first user directive including at least one first model requirement;
determining, based at least in part on the first user directive, a prior user directive associated with one or more prior model requirements, the one or more prior model requirements and the at least one first model requirement including a common requirement, wherein the common requirement includes at least one of a common target feature or a common raw data source, and wherein determining the prior use directive comprises:
comparing the at least one first model requirement with at least one prior model requirement of the one or more prior model requirements,
determining, based at least in part on the comparing of the at least one first model requirement and the at least one prior model requirement, a similarity score between the first user directive and the prior user directive, and
determining the prior user directive based at least in part on the similarity score;
identifying one or more features associated with the prior user directive;
selecting at least one selected feature of the one or more features;
generating, in response to the receiving of the first user directive, a first model using the at least one selected feature;
determining a performance of the first model;
and
monitoring, via an at least partially automated process, the performance of the model to determine changes in model performance over a period of time.

12 more elements

One or more non-transitory computer-readable media maintaining instructions executable by one or more processors to perform operations comprising:

receiving model input data and model output data of a model, wherein the model input data includes at least one of customer behavior data or a raw data source;
determining a distribution of the model input data and of the model output data over a first period of time, wherein the first period of time is predetermined based at least in part on a user directive associated with the model;
determining, based on the distribution of the model input data and of the model output data, a first distribution score associated with the model input data and a second distribution score associated with the model output data;
determining a second period of time based at least in part on at least one of the first distribution score or the second distribution score;
monitoring, via an at least a partially automated process, changes in the first distribution score and in the second distribution score over the second period of time;
and
retraining the model based at least in part on a determination that the changes of at least one of the first distribution score or the second distribution score are outside of a threshold distribution range.

4 more elements

Dependencies are inferred from the claim text.