US 10,719,301 B1Granted

Development environment for machine learning media models

AssigneeAmazon Technologies, Inc.

InventorsSunny Dasgupta, Sri Kaushik Pavani, Shriram Venkataramana, Sabya Sachi, Himanshu Prafulla Shringarpure, Divya Varshney, FNU Najih, Suryansh Purwar, Niyaz Puzhikkunnath, Mathew Philip, Shubhangam Agrawal

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Computer systems and associated methods are disclosed to implement a model development environment (MDE) that allows a team of users to perform iterative model experiments to develop machine learning (ML) media models. In embodiments, the MDE implements a media data management interface that allows users to annotate and manage training data for models. In embodiments, the MDE implements a model experimentation interface that allows users to configure and run model experiments, which include a training run and a test run of a model. In embodiments, the MDE implements a model diagnosis interface that displays the model's performance metrics and allows users to visually inspect media samples that were used during the model experiment to determine corrective actions to improve model performance for later iterations of experiments. In embodiments, the MDE allows different types of users to collaborate on a series of model experiments to build an optimal media model.

Claims20

Dependent claims

A system comprising:

one or more computers that implement a model development environment (MDE) for developing machine learning (ML) image models, configured to:
generate a first user interface configured for a user to:
annotate a training data set of images for a ML image model, wherein the ML image model is configured to make predictions about images;
generate a second user interface configured for the user to perform a model experiment to:
execute a training run of the ML image model using the training data set as modified according to user input via the first user interface;
and
execute a test run of the ML image model after the training run using a test data set and generate performance metrics for the ML image model from the test run;
generate a third user interface configured to display the performance metrics and for the user to:
specify one or more further modifications to the training data set to improve prediction performance for a next iteration of the ML image model to be trained and tested in a next model experiment;
deploy an approved iteration of the ML image model to a production environment as a production model;
obtain production input data and prediction results for the production model and generate additional performance metrics for the production model from the production input data and prediction results;
and
responsive to a detection of a performance aberration in the additional performance metrics, generate a fourth user interface to report the performance aberration.

10 more elements

A computer-implemented method comprising:

performing, by a model development environment (MDE) implemented on one or more processors and associated memory:
generating a first user interface configured for a user to:
annotate a training data set of media samples for a machine learning (ML) model, wherein the ML model is configured to make predictions about the media samples;
generating a second user interface configured for the user to perform a model experiment to:
cause a training run of the ML model to be executed using the training data set as modified according to user input via the first user interface;
and
cause a test run of the ML model to be executed after the training run using a test data set and generate performance metrics for the ML model from the test run;
generating a third user interface displaying the performance metrics and configured for the user to:
specify one or more further modifications to the training data set to improve prediction performance for a next iteration of the ML model to be trained and tested in a next model experiment.

6 more elements

One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors implement a model development environment (MDE) and cause the MDE to:

generate a first user interface configured for a user to:
annotate a training data set of media samples for a machine learning (ML) model, wherein the ML model is configured to make predictions about the media samples;
generate a second user interface configured for the user to:
cause a training run of the ML model to be executed using the training data set as modified according to user input via the first user interface;
and
cause a test run of the ML model to be executed after the training run and generate performance metrics for the ML model from the test run;
generate a third user interface displaying the performance metrics and configured for the user to:
specify one or more further modifications to the training data set to improve prediction performance for a next iteration of the ML model to be trained and tested in a next model experiment.

5 more elements

Dependencies are inferred from the claim text.