Back to results
US 11,636,286 B1Granted
Concurrent ensemble model training for open sets
AssigneeAmazon Technologies, Inc.
InventorsRoman Goldenberg, Miriam Farber, George Leifman, Gerard Guy Medioni
Life of this patent
Estimating the term…
Abstract
Described are systems and methods for training machine learning models of an ensemble of models that are de-correlated. For example, two or more machine learning models may be concurrently trained (e.g., co-trained) while adding a decorrelation component to one or both models that decreases the pairwise correlation between the outputs of the models. Unlike traditional approaches, in accordance with the disclosed implementations, only the negative results need to be decorrelated.
Claims20
A computer-implemented method to train each of a plurality of machine learning models of an ensemble of machine learning models, comprising:
- initiating a first training of a first machine learning model of a plurality of machine learning models using a training data that includes a plurality of training images through which the ensemble of the plurality of machine learning models is to be trained for a plurality of classes, wherein the first machine learning model is trained with an initial loss function;
- initiating, concurrent with the first training of the first machine learning model, a second training of a second machine learning model of the plurality of machine learning models using the training data, wherein the second machine learning model is trained with the initial loss function;
- determining, for the first machine learning model, a first probability vector for a training image, wherein the first probability vector indicates a first probability for each of the plurality of classes that the training image corresponds with the class;
- extracting, based at least in part on the first probability vector, a first individual loss for the first machine learning model;
- determining, for the second machine learning model, a second probability vector for the training image, wherein the second probability vector indicates a second probability for each of the plurality of classes that the training image corresponds with the class;
- extracting, based at least in part on the second probability vector, a second individual loss for the second machine learning model;
- creating, based at least in part on the first probability vector and the second probability vector, a combined decorrelation component that is indicative of the first probability vector and the second probability vector for classes of the plurality of classes that do not correspond to the training image and excludes a correct class that corresponds to the training image;
- updating the initial loss function to include the first individual loss, the second individual loss, and the combined decorrelation component to create an updated loss function;
- and
- continuing the concurrent training of the first machine learning model and the second machine learning model with the updated loss function to produce a first trained machine learning model and a second trained machine learning model of the ensemble.
7 more elements
A computing system, comprising:
- one or more processors;
- and
- a memory storing program instructions that when executed by the one or more processors cause the one or more processors to at least:
- determine, as part of a concurrent training of a first machine learning model of a plurality of machine learning models of an ensemble and a second machine learning model of the plurality of machine learning models of the ensemble:
- a first probability vector generated by the first machine learning model with respect to a training data sample, wherein the first probability vector indicates a first probability for each of a plurality of classes that the training data sample corresponds with the class;
- and
- a second probability vector generated by the second machine learning model with respect to the training data sample, wherein the second probability vector indicates a second probability for each of the plurality of classes that the training data sample corresponds with the class;
- create, based at least in part on the first probability vector and the second probability vector, a combined decorrelation component that is indicative of the first probability vector and the second probability vector for classes of the plurality of classes that do not correspond to the training data sample and excludes a correct class that corresponds to the training data sample;
- update a loss function to include the combined decorrelation component to create an updated loss function;
- and
- continue, using the updated loss function, the concurrent training of the first machine learning model of the ensemble and the second machine learning model of the ensemble to produce a first trained machine learning model and a second trained machine learning model of the ensemble.
8 more elements
A computer-implemented method, comprising:
- concurrently training a plurality of machine learning models of an ensemble of machine learning models to classify data samples of an open set of data samples;
- determining, during the concurrent training of each of the plurality of machine learning models of the ensemble, a combined decorrelation component indicative of probabilities determined by each of the plurality of machine learning models for a plurality of classes that do not correspond to a training data sample;
- updating a loss function used to concurrently train each of the plurality of machine learning models to include the combined decorrelation component;
- and
- continuing the concurrent training of the plurality of machine learning models of the ensemble to produce a plurality of trained machine learning models.
2 more elements
Dependencies are inferred from the claim text.