# US 2024/0005199 A1 · Automated concept drift detection in live machine learning systems for machine learning model updating

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0005199 A1
- Document ID: US20240005199A1
- Status: Published application
- Kind: A1
- Application number: 17/853,189
- Filed: 2022-06-29
- Published: 2024-01-04
- Earliest claimed: 2022-06-29
- Assignees: Actimize Ltd.
- Applicants: Actimize LTD.
- Inventors: Danny BUTVINIK; Yoav AVNEON; Elina MALIARSKY
- Web page: https://patentgrep.com/patent/US20240005199A1

## CPC

- G06N 20/00: Machine learning
- G06F 18/2193: {based on specific statistical tests}
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06Q 20/4016: {involving fraud or risk level assessment in transaction processing}
- G06Q 40/02: Banking, e.g. interest calculation or account maintenance (credit or loans G06Q40/03)
- G06Q 40/06: Asset management; Financial planning or analysis
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

A machine learning (ML) system and methods are provided that are configured to detect concept drift in ML models. The system includes a processor and a computer readable medium operably coupled thereto, the computer readable medium comprising a plurality of instructions stored in association therewith that are accessible to, and executable by, the processor, to perform drift detection operations which include receiving a first data set for use during online training of a first ML model, determining a change to an uncertainty bound metric associated with classifiers for features utilized by the first ML model, identifying that the first data set causes the concept drift with the online training of the first ML model, determining characterization information about a type of the concept drift, generating an ML update paradigm based on the concept drift and the characterization information, alerting an ML model updater of the ML update paradigm.

## Claims

### Claim 1

A machine learning (ML) system configured to detect concept drift in ML models during ML model operations using live data, the ML system comprising:
a processor and a computer readable medium operably coupled thereto, the computer readable medium comprising a plurality of instructions stored in association therewith that are accessible to, and executable by, the processor, to perform drift detection operations which comprise:
receiving a first data set associated with the live data for use during online training of a first ML model of the ML models;
determining a change to an uncertainty bound metric associated with classifiers for features utilized by the first ML model based on the first data set and at least one second data set from previous data utilized with training the first ML model;
identifying that the first data set causes the concept drift with the online training of the first ML model based on the change to the uncertainty bound metric;
determining characterization information about a type of the concept drift based at least on the change to the uncertainty bound metric, wherein the characterization information is associated with the features utilized by the first ML model;
generating an ML update paradigm based on the concept drift of the first ML model and the characterization information; and
alerting an ML model updater of the ML update paradigm.

### Claim 2 (depends on 1)

The ML system of claim 1, wherein determining the change to the uncertainty bound metric comprises:
calculating a SHAP Additive exPlanation (SHAP) value of each of the features utilized by the first ML model for a corresponding one of the classifiers;
calculating comparisons of the SHAP values of the features to previous SHAP values of the features to the classifiers of the first ML model; and
calculating covariances between the SHAP values and the previous SHAP values using the comparisons,
wherein the change is determined based on the calculated comparisons and the calculated covariances.

### Claim 3 (depends on 2)

The ML system of claim 2, wherein the drift detection operations further comprise:
determining that the change to the uncertainty bound metric exceeds a threshold change, wherein threshold change indicates a degradation in performance of the first ML model or a fluctuation in model performance of the first ML model,
wherein the characterization information comprises at least one of a type or a degree of the concept drift based on the SHAP values and the change exceeding the threshold change.

### Claim 4 (depends on 3)

The ML system of claim 3, wherein the change comprises a delta change value for margin densities between the first data set associated with the live data and the at least one second data set from the previous data.

### Claim 5 (depends on 2)

The ML system of claim 2, wherein the drift detection operations comprise:
determining at least one of a shape, a curvature, or a slope of each of the covariances; and
determining the concept drift based on the at least one of the shape, the curvature, or the slope of each of the covariances,
wherein the first data set is identified as causing the concept drift based on determining the concept drift.

### Claim 6 (depends on 5)

The ML system of claim 5, wherein determining the characterization information comprises:
generating a model explanation output based on at least one of the SHAP values or a ranking of each of the features for each of the classifiers of the first ML model,
wherein the characterization information comprises the model explanation output.

### Claim 7 (depends on 1)

The ML system of claim 1, wherein the ML models comprise a plurality of fraud assessment models for a risk assessment engine utilizing the ML models with a financial system.

### Claim 8 (depends on 1)

The ML system of claim 1, wherein the drift detection operations further comprise:
generating a recommended adjustment to at least one of the first ML model or a second ML model for the concept drift based on at least one of the ML update paradigm or the change to the uncertainty bound metric; and
outputting the recommended adjustment to the ML model updater.

### Claim 9

A method to detect concept drift in ML models during ML model operations using live data, the method comprising:
receiving a first data set associated with the live data for use during online training of a first ML model of the ML models;
determining a change to an uncertainty bound metric associated with classifiers for features utilized by the first ML model based on the first data set and at least one second data set from previous data utilized with training the first ML model;
identifying that the first data set causes the concept drift with the online training of the first ML model based on the change to the uncertainty bound metric;
determining characterization information about a type of the concept drift based at least on the change to the uncertainty bound metric, wherein the characterization information is associated with the features utilized by the first ML model;
generating an ML update paradigm based on the concept drift of the first ML model and the characterization information; and
alerting an ML model updater of the ML update paradigm.

### Claim 10 (depends on 9)

The method of claim 9, wherein determining the change to the uncertainty bound metric comprises:
calculating a SHAP Additive exPlanation (SHAP) value of each of the features utilized by the first ML model for a corresponding one of the classifiers;
calculating comparisons of the SHAP values of the features to previous SHAP values of the features to the classifiers of the first ML model; and
calculating covariances between the SHAP values and the previous SHAP values using the comparisons,
wherein the change is determined based on the calculated comparisons and the calculated covariances.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
determining that the change to the uncertainty bound metric exceeds a threshold change, wherein threshold change indicates a degradation in performance of the first ML model or a fluctuation in model performance of the first ML model,
wherein the characterization information comprises at least one of a type or a degree of the concept drift based on the SHAP values.

### Claim 12 (depends on 11)

The method of claim 11, wherein the change comprises a delta change value for margin densities between the first data set associated with the live data and the at least one second data set from the previous data.

### Claim 13 (depends on 10)

The method of claim 10, further comprising:
determining at least one of a shape, a curvature, or a slope of each of the covariances; and
determining the concept drift based on the at least one of the shape, the curvature, or the slope of each of the covariances,
wherein the first data set is identified as causing the concept drift based on determining the concept drift.

### Claim 14 (depends on 13)

The method of claim 13, wherein determining the characterization information comprises:
generating a model explanation output based on at least one of the SHAP values or a ranking of each of the features for each of the classifiers of the first ML model,
wherein the characterization information comprises the model explanation output.

### Claim 15 (depends on 9)

The method of claim 9, wherein the ML models comprise a plurality of fraud assessment models for a risk assessment engine utilizing the ML models with a financial system.

### Claim 16 (depends on 9)

The method of claim 9, further comprising:
generating a recommended adjustment to at least one of the first ML model or a second ML model for the concept drift based on at least one of the ML update paradigm or the change to the uncertainty bound metric; and
outputting the recommended adjustment to the ML model updater.

### Claim 17

A non-transitory computer-readable medium having stored thereon computer-readable instructions executable to detect concept drift in ML models during ML model operations using live data, the computer-readable instructions executable to perform drift detection operations which comprises:
receiving a first data set associated with the live data for use during online training of a first ML model of the ML models;
determining a change to an uncertainty bound metric associated with classifiers for features utilized by the first ML model based on the first data set and at least one second data set from previous data utilized with training the first ML model;
identifying that the first data set causes the concept drift with the online training of the first ML model based on the change to the uncertainty bound metric;
determining characterization information about a type of the concept drift based at least on the change to the uncertainty bound metric, wherein the characterization information is associated with the features utilized by the first ML model;
generating an ML update paradigm based on the concept drift of the first ML model and the characterization information; and
alerting an ML model updater of the ML update paradigm.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein determining the change to the uncertainty bound metric comprises:
calculating a SHAP Additive exPlanation (SHAP) value of each of the features utilized by the first ML model for a corresponding one of the classifiers;
calculating comparisons of the SHAP values of the features to previous SHAP values of the features to the classifiers of the first ML model; and
calculating covariances between the SHAP values and the previous SHAP values using the comparisons,
wherein the change is determined based on the calculated comparisons and the calculated covariances.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the drift detection operations further comprise:
determining that the change to the uncertainty bound metric exceeds a threshold change, wherein threshold change indicates a degradation in performance of the first ML model or a fluctuation in model performance of the first ML model,
wherein the characterization information comprises at least one of a type or a degree of the concept drift based on the SHAP values.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the change comprises a delta change value for margin densities between the first data set associated with the live data and the at least one second data set from the previous data.
