US 12,586,566 B1Granted

Model drift detection techniques

AssigneeAmazon Technologies, Inc.

InventorsAli Takbiriborujeni, Jainik Somabhai Patel, Gaurav Gada, Tianqi Wu, Sagar Chaturvedi

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques for detecting machine learning model drift are described. Model drift can result in the model misclassifying inputs. A system for detecting drift in natural language processing (NLP) models involves determining high-dimensional embeddings of inputs and high-dimensional embeddings of training samples, reducing the high-dimensional embeddings to low-dimensional embeddings, and comparing the low-dimensional embeddings to determine whether the inputs are statistically different than the training samples. When the inputs are statistically different than the training samples, model drift is detected, and retraining of the model may be performed. The system can detect drift in other classification models as well and can process with respect to other types of inputs (e.g., audio, image, etc.).

Claims18

Dependent claims

A computer-implemented method, comprising:

receiving a plurality of inputs;
processing the plurality of inputs to determine first embedding data corresponding to the plurality of inputs, the first embedding data including first feature vectors having a first dimensionality;
processing, using a machine learning model, the first embedding data to classify the plurality of inputs;
determining that a first subset of the plurality of inputs correspond to a first class;
processing the first subset of the plurality of inputs to determine second embedding data corresponding to the first subset of the plurality of inputs, the second embedding data including second feature vectors having the first dimensionality;
processing the second embedding data to determine first reduced embedding data corresponding to the second embedding data, the first reduced embedding data including third feature vectors having a second dimensionality, the second dimensionality being smaller than the first dimensionality;
receiving a first plurality of training samples corresponding to the first class, the first plurality of training samples having been used to train the machine learning model;
processing the first plurality of training samples to determine third embedding data corresponding to the first plurality of training samples, the third embedding data including fourth feature vectors having the first dimensionality;
processing the third embedding data to determine second reduced embedding data corresponding to the third embedding data, the second reduced embedding data including fifth feature vectors having the second dimensionality;
determining that differences between the first reduced embedding data and the second reduced embedding data satisfy one or more conditions;
and
based on the differences satisfying the one or more conditions, generating a first indicator representing that the machine learning model likely misclassified one or more of the first subset of the plurality of inputs.

9 more elements

A system, comprising:

at least one processor;
and
at least one memory including instructions that, when executed by the at least one processor, cause the system to:
receive a plurality of inputs;
process the plurality of inputs to determine first embedding data corresponding to the plurality of inputs, the first embedding data including first feature vectors having a first dimensionality;
process, using a machine learning model, the first embedding data to classify the plurality of inputs;
determine that a first subset of the plurality of inputs correspond to a first class;
process the first subset of the plurality of inputs to determine second embedding data corresponding to the first subset of the plurality of inputs, the second embedding data including second feature vectors having the first dimensionality;
process the second embedding data to determine first reduced embedding data corresponding to the second embedding data, the first reduced embedding data including third feature vectors having a second dimensionality, the second dimensionality being smaller than the first dimensionality;
receive a first plurality of training samples corresponding to the first class, the first plurality of training samples having been used to train the machine learning model;
process the first plurality of training samples to determine third embedding data corresponding to the first plurality of training samples, the third embedding data including fourth feature vectors having the first dimensionality;
process the third embedding data to determine second reduced embedding data corresponding to the third embedding data, the second reduced embedding data including fifth feature vectors having the second dimensionality;
determine that differences between the first reduced embedding data and the second reduced embedding data satisfy one or more conditions;
and
based on the differences satisfying the one or more conditions, generate a first indicator representing that the machine learning model likely misclassified one or more of the first subset of the plurality of inputs.

12 more elements

Dependencies are inferred from the claim text.