# US 2021/0334695 A1 · System to correct model drift in machine learning application

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334695 A1
- Document ID: US20210334695A1
- Status: Published application
- Kind: A1
- Application number: 16/859,684
- Filed: 2020-04-27
- Published: 2021-10-28
- Earliest claimed: 2020-04-27
- Assignees: Bank of America Corporation
- Applicants: Bank of America Corporation
- Inventors: Utkarsh Raj; Maharaj Mukherjee
- Web page: https://patentgrep.com/patent/US20210334695A1

## CPC

- G06N 20/00: Machine learning
- G06F 11/3409: {for performance assessment}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A model correction tool automatically detects and corrects model drift in a model for a machine learning application. To detect drift, the tool continuously monitors input data, outputs, and/or technical resources (e.g., processor, memory, network, and input/output resources) used to generate outputs. The tool analyzes changes to input data, outputs, and/or resource usage to determine when drift has occurred. When drift is determined to be occurring, the tool retrains a model for a machine learning application.

## Claims

### Claim 1

An apparatus comprising:
a memory; and
a hardware processor communicatively coupled to the memory, the hardware processor configured to:
receive an input data into a machine learning application;
compare the input data with previous input data into the machine learning application to determine a change in a statistical property of the input data;
receive an output prediction of the machine learning application made by applying a model on the input data;
determine an accuracy of the model by comparing the output with previous outputs generated by the model;
determine, based on the change in the statistical property of the input data and the accuracy of the model, a drift score for the model; and
retrain the model when the drift score exceeds a threshold.

### Claim 2 (depends on 1)

The apparatus of claim 1, the hardware processor further configured to:
receive an indication of computing resources used by the machine learning application to make the output prediction based on the input data; and
compare the indication of computing resources with previous indications of computing resources used when applying the model to determine a change in computing resources used when applying the model, the drift score is further based on the change in computing resources used when applying the model.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the computing resources comprises at least one of processor resources and memory resources.

### Claim 4 (depends on 2)

The apparatus of claim 2, wherein retraining the model optimizes the computing resources used when applying the model.

### Claim 5 (depends on 1)

The apparatus of claim 1, the hardware processor further configured to determine a time series distribution of the input data and the previous input data, the change in the input data is further determined based on the time series distribution.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the drift score is determined by calculating a weighted average of the change in the statistical property of the input data and the accuracy of the model.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein retraining the model when the drift score exceeds the threshold occurs automatically as part of a model governance framework.

### Claim 8

A method comprising:
receiving, by a hardware processor communicatively coupled to a memory, an input data into a machine learning application;
comparing, by the hardware processor, the input data with previous input data into the machine learning application to determine a change in a statistical property of the input data;
receiving, by the hardware processor, an output prediction of the machine learning application made by applying a model on the input data;
determining, by the hardware processor, an accuracy of the model by comparing the output with previous outputs generated by the model;
determining, by the hardware processor and based on the change in the statistical property of the input data, the accuracy of the model, and the change in processor resources and memory resources used when applying the model, a drift score for the model; and
retraining, by the hardware processor, the model when the drift score exceeds a threshold.

### Claim 9 (depends on 8)

The method of claim 8, further comprising:
receiving, by the hardware processor, an indication of computing resources used by the machine learning application to make the output prediction based on the input data;
comparing, by the hardware processor, the indication of computing resources with previous indications of computing resources used when applying the model to determine a change in computing resources used when applying the model, the drift score is further based on the change in computing resources used when applying the model.

### Claim 10 (depends on 9)

The method of claim 9, wherein the computing resources comprise at least one of processor resources and memory resources.

### Claim 11 (depends on 9)

The method of claim 9, wherein retraining the model optimizes the computing resources used when applying the model.

### Claim 12 (depends on 8)

The method of claim 8, further comprising determining, by the hardware processor, a time series distribution of the input data and the previous input data, the change in the input data is further determined based on the time series distribution.

### Claim 13 (depends on 8)

The method of claim 8, wherein the drift score is determined by calculating a weighted average of the change in the statistical property of the input data and the accuracy of the model.

### Claim 14 (depends on 8)

The method of claim 8, wherein retraining the model when the drift score exceeds the threshold occurs automatically as part of a model governance framework.

### Claim 15

A system comprising:
a device; and
a model correction tool comprising:
a memory; and
a hardware processor communicatively coupled to the memory, the hardware processor configured to:
receive, from the device, an input data into a machine learning application;
compare the input data with previous input data into the machine learning application to determine a change in a statistical property of the input data;
receive an output prediction of the machine learning application made by applying a model on the input data;
determine an accuracy of the model by comparing the output with previous outputs generated by the model;
determine, based on the change in the statistical property of the input data, the accuracy of the model, and the change in processor resources and memory resources used when applying the model, a drift score for the model; and
retrain the model when the drift score exceeds a threshold.

### Claim 16 (depends on 15)

The system of claim 15, the hardware processor further configured to:
receive an indication of computing resources used by the machine learning application to make the output prediction based on the input data;
compare the indication of computing resources with previous indications of computing resources used when applying the model to determine a change in computing resources used when applying the model, the drift score is further based on the change in computing resources used when applying the model.

### Claim 17 (depends on 16)

The system of claim 16, wherein the computing resources comprises at least one of processor resources and memory resources.

### Claim 18 (depends on 16)

The system of claim 16, wherein retraining the model optimizes the computing resources used when applying the model.

### Claim 19 (depends on 15)

The system of claim 15, the hardware processor further configured to determine a time series distribution of the input data and the previous input data, the change in the input data is further determined based on the time series distribution.

### Claim 20 (depends on 15)

The system of claim 15, wherein the drift score is determined by calculating a weighted average of the change in the statistical property of the input data, the accuracy of the model, and the change in processor resources and memory resources used when applying the model.
