# US 2025/0165375 A1 · Detecting model deviation with inter-group metrics

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0165375 A1
- Document ID: US20250165375A1
- Status: Published application
- Kind: A1
- Application number: 18/949,310
- Filed: 2024-11-15
- Published: 2025-05-22
- Earliest claimed: 2023-11-17
- Assignees: The Toronto-Dominion Bank
- Applicants: The Toronto-Dominion Bank
- Inventors: Jesse Cole Cresswell; George Frazer Stein; Farnush Farhadi Hassan Kiadeh; Zhaoyan Liu; Ji Xin
- Web page: https://patentgrep.com/patent/US20250165375A1

## CPC

- G06N 20/00: Machine learning
- G06F 11/3447: {Performance evaluation by modeling}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer model is monitored during operation to evaluate performance of the model with respect to different groups evaluated by the model. Performance for each group is evaluated to determine an inter-group performance metric describing how model predictions across groups differs. A threshold for excess inter-group performance differences can be calibrated using withheld training data or out-of-time data to provide a statistical guarantee for detecting meaningful variation in inter-group performance metric differences. When the inter-group performance exceeds the threshold, the computer model may be considered to deviate from expected behavior and the monitoring can act to correct its operation, for example, by modifying actions that may otherwise occur due to model predictions or by initiating model retraining.

## Claims

### Claim 1

A system for detecting inter-group performance deviation of a computer model, comprising:
a processor configured to execute instructions; and
a non-transitory computer-readable memory having a set of instructions executable by the processor for:
identifying a set of model predictions for a set of data samples applied to a trained computer model, each data sample being associated with at least one group of a plurality of groups;
determining a plurality of group performance metrics, each corresponding to one of the plurality of groups based on the model predictions for data samples associated with the respective group;
determining an inter-group performance metric for the computer model based on the plurality of group performance metrics;
determining that the inter-group performance metric exceeds an inter-group performance threshold calibrated based on a plurality of calibration inter-group performance metrics; and
responsive to the determination that the inter-group performance metrics exceeds the inter-group performance threshold, retraining at least one parameter of the computer model to reduce the inter-group performance metric.

### Claim 2 (depends on 1)

The system of claim 1, wherein retraining at least one parameter of the computer model comprises modifying an activation function.

### Claim 3 (depends on 1)

The system of claim 1, wherein retraining at least one parameter of the computer model comprises training the computer model with a training data set including the set of data samples.

### Claim 4 (depends on 1)

The system of claim 1, wherein the inter-group performance metric is a difference in false positive rate.

### Claim 5 (depends on 1)

The system of claim 1, wherein the inter-group performance metric is determined for a plurality of sequential time periods and retraining the at least one parameter of the computer model is performed when the inter-group performance metric is exceeded for the plurality of sequential time periods.

### Claim 6 (depends on 1)

The system of claim 1, wherein the plurality of calibration inter-group performance metrics are determined based on one or more data sets associated with time periods earlier than the set of data samples.

### Claim 7 (depends on 1)

The system of claim 1, the instructions further comprising calibrating the inter-group performance threshold by:
estimating a probability distribution for the plurality of calibration inter-group performance metrics; and
setting the inter-group performance threshold to a threshold percentile of the probability distribution.

### Claim 8 (depends on 1)

The system of claim 1, the instructions further comprising calibrating the inter-group performance threshold by:
determining a first subset and a second subset of data samples of a calibration data set by sampling from the calibration data set;
determining a first calibration inter-group performance metric for the first subset and a second calibration inter-group performance metric for the second subset, the first and second calibration inter-group performance metrics included in the plurality of inter-group performance metrics; and
setting the inter-group performance threshold based on the plurality of inter-group performance metrics.

### Claim 9

A method for detecting inter-group performance deviation of a computer model, comprising:
identifying a set of model predictions for a set of data samples applied to a trained computer model, each data sample being associated with at least one group of a plurality of groups;
determining a plurality of group performance metrics, each corresponding to one of the plurality of groups based on the model predictions for data samples associated with the respective group;
determining an inter-group performance metric for the computer model based on the plurality of group performance metrics;
determining that the inter-group performance metric exceeds an inter-group performance threshold calibrated based on a plurality of calibration inter-group performance metrics; and
responsive to the determination that the inter-group performance metrics exceeds the inter-group performance threshold, retraining at least one parameter of the computer model to reduce the inter-group performance metric.

### Claim 10 (depends on 9)

The method of claim 9, wherein retraining at least one parameter of the computer model comprises modifying an activation function.

### Claim 11 (depends on 9)

The method of claim 9, wherein retraining at least one parameter of the computer model comprises training the computer model with a training data set including the set of data samples.

### Claim 12 (depends on 9)

The method of claim 9, wherein the inter-group performance metric is a difference in false positive rate.

### Claim 13 (depends on 9)

The method of claim 9, wherein the inter-group performance metric is determined for a plurality of sequential time periods and retraining the at least one parameter of the computer model is performed when the inter-group performance metric is exceeded for the plurality of sequential time periods.

### Claim 14 (depends on 9)

The method of claim 9, wherein the plurality of calibration inter-group performance metrics are determined based on one or more data sets associated with time periods earlier than the set of data samples.

### Claim 15 (depends on 9)

The method of claim 9, wherein the method further comprises calibrating the inter-group performance threshold by:
estimating a probability distribution for the plurality of calibration inter-group performance metrics; and
setting the inter-group performance threshold to a threshold percentile of the probability distribution.

### Claim 16 (depends on 9)

The method of claim 9, wherein the method further comprises calibrating the inter-group performance threshold by:
determining a first subset and a second subset of data samples of a calibration data set by sampling from the calibration data set;
determining a first calibration inter-group performance metric for the first subset and a second calibration inter-group performance metric for the second subset, the first and second calibration inter-group performance metrics included in the plurality of inter-group performance metrics; and
setting the inter-group performance threshold based on the plurality of inter-group performance metrics.

### Claim 17

A non-transitory computer-readable medium, the non-transitory computer-readable medium comprising instructions executable by a processor for:
identifying a set of model predictions for a set of data samples applied to a trained computer model, each data sample being associated with at least one group of a plurality of groups;
determining a plurality of group performance metrics, each corresponding to one of the plurality of groups based on the model predictions for data samples associated with the respective group;
determining an inter-group performance metric for the computer model based on the plurality of group performance metrics;
determining that the inter-group performance metric exceeds an inter-group performance threshold calibrated based on a plurality of calibration inter-group performance metrics; and
responsive to the determination that the inter-group performance metrics exceeds the inter-group performance threshold, retraining at least one parameter of the computer model to reduce the inter-group performance metric.

### Claim 18 (depends on 17)

The computer-readable medium of claim 17, wherein retraining at least one parameter of the computer model comprises modifying an activation function.

### Claim 19 (depends on 17)

The computer-readable medium of claim 17, wherein retraining at least one parameter of the computer model comprises training the computer model with a training data set including the set of data samples.

### Claim 20 (depends on 17)

The computer-readable medium of claim 17, wherein the inter-group performance metric is a difference in false positive rate.
