US 2025/0028879 A1Published application

Systems and methods for forecasting data drift for model monitoring

AssigneeCapital One Services, LLC

InventorsBrian BARR, Jeremy GOODSITT

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and methods for forecasting data drift for model monitoring. In some aspects, the system receives a current explainability vector for a machine learning model and a data drift vector for historical data profiles. The machine learning model is trained on historical data including values for a first set of features. The system generates a projected synthetic dataset using the data drift vector and updates the machine learning model based on the projected synthetic dataset. Using the current explainability vector and a future explainability vector for the updated model, the system generates a second set of features and determines a drift threshold vector for the second set of features based on values in the explainability vectors. The system determines a discrepancy score for each feature of the second set of features. The system generates an alert including features in the second set of features and their associated discrepancy scores.

Claims20

Dependent claims

A system for using historical data profiles, projected synthetic datasets, and explainable artificial intelligence techniques to forecast data drift for model monitoring, comprising:

one or more processors;
one or more non-transitory computer-readable media comprising instructions that, when executed by the one or more processors, cause operations comprising:
receiving a machine learning model and a plurality of historical data profiles, wherein the machine learning model is trained on historical data, wherein the historical data profiles correspond to instances of the historical data at different times, and wherein the historical data comprises values for a first set of features;
processing the machine learning model to extract a current explainability vector, wherein each entry in the current explainability vector corresponds to a feature in the first set of features and is indicative of a correlation between the feature and an output of the machine learning model;
based on the plurality of historical data profiles, generating a data drift vector, wherein each entry in the data drift vector is indicative of expected change to a feature of the first set of features for the historical data;
using the data drift vector, generating a projected synthetic dataset, wherein the projected synthetic dataset comprises values for the first set of features at a future time;
updating the machine learning model based on the projected synthetic dataset to generate an updated machine learning model;
processing the updated machine learning model to extract a future explainability vector, wherein each entry in the future explainability vector corresponds to a feature in the first set of features and is indicative of a correlation between the feature and an output of the updated machine learning model;
using the current explainability vector and the future explainability vector, generating a second set of features including features with associated values in the current explainability vector or the future explainability vector above a threshold, wherein the second set of features is a subset of the first set of features;
determining a drift threshold vector for the second set of features based on an associated value for each feature in the current explainability vector and the future explainability vector, wherein each entry in the drift threshold vector corresponds to a feature of the second set of features and is indicative of a degree of allowed drift for the feature;
based on an associated entry in the drift threshold vector and the data drift vector, for each feature of the second set of features, determining a discrepancy score between the associated entry in the drift threshold vector and an associated entry in the data drift vector;
and
generating a notification dashboard, comprising an alert including one or more features in the second set of features and their associated discrepancy scores.

10 more elements

A method for using historical data profiles, projected synthetic datasets, and explainable artificial intelligence techniques to forecast data drift for model monitoring, comprising:

receiving a current explainability vector for a machine learning model and a data drift vector for a plurality of historical data profiles, wherein the machine learning model is trained on historical data, wherein the historical data profiles correspond to instances of the historical data at different times, and wherein the historical data comprises values for a first set of features;
using the data drift vector, generating a projected synthetic dataset, wherein the projected synthetic dataset comprises values for the first set of features at a future time;
updating the machine learning model based on the projected synthetic dataset to generate an updated machine learning model;
using the current explainability vector and a future explainability vector for the updated machine learning model, generating a second set of features, wherein the second set of features is a subset of the first set of features;
determining a drift threshold vector for the second set of features based on an associated value for each feature in the current explainability vector and the future explainability vector;
based on an associated entry in the drift threshold vector and the data drift vector, for each feature of the second set of features, determining a discrepancy score between the associated entry in the drift threshold vector and an associated entry in the data drift vector;
and
generating an alert including one or more features in the second set of features and their associated discrepancy scores.

5 more elements

One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause operations comprising:

using a data drift vector for a plurality of historical data profiles, generating a projected synthetic dataset, wherein the projected synthetic dataset comprises values for a first set of features at a future time, wherein the plurality of historical data profiles correspond to instances of historical data at different times, and wherein the historical data comprises values for the first set of features;
using a current explainability vector for a machine learning model trained on the historical data and a future explainability vector for an updated machine learning model generated from updating the machine learning model on the projected synthetic dataset, generating a second set of features;
based on an associated entry in a drift threshold vector for the second set of features and the data drift vector, for each feature of the second set of features, determining a discrepancy score between the associated entry in the drift threshold vector and an associated entry in the data drift vector;
and
generating an alert including one or more features in the second set of features and their associated discrepancy scores.

2 more elements

Dependencies are inferred from the claim text.