# US 2022/0012613 A1 · System and method for evaluating machine learning model behavior over data segments

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0012613 A1
- Document ID: US20220012613A1
- Status: Published application
- Kind: A1
- Application number: 17/371,551
- Filed: 2021-07-09
- Published: 2022-01-13
- Earliest claimed: 2020-07-09
- Assignees: Truera, Inc.
- Applicants: Truera, Inc.
- Inventors: Anupam Datta; Shayak Sen; Apoorv Gupta; David Sandai Kurokawa
- Web page: https://patentgrep.com/patent/US20220012613A1

## CPC

- G06N 5/041: {Abduction}
- G06N 20/20: Ensemble learning
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computing machine receives a representation of a machine learning model, a representation of a first data segment, and a representation of a second data segment. The computing machine computes an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment. The computing machine determines a set of reasons for the computed output difference based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference. The computing machine provides an output representing the set of reasons.

## Claims

### Claim 1

A method comprising:
receiving, at a computing machine, a representation of a machine learning model, a representation of a first data segment, and a representation of a second data segment;
computing, at the computing machine, an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment;
determining, using the computing machine, a set of reasons for the computed output difference based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference; and
providing an output representing the set of reasons.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
generating, using the computing machine, a remediation engine for the machine learning model to decrease a magnitude of the output difference, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features.

### Claim 3 (depends on 2)

The method of claim 2, wherein the remediation engine selects features for adjustment based on a user input received at the computing machine.

### Claim 4 (depends on 1)

The method of claim 1, wherein the set of metrics comprises a metric based on a Difference of Means or a Wasserstein Distance.

### Claim 5 (depends on 1)

The method of claim 1, wherein the output difference comprises a statistical difference measurement.

### Claim 6 (depends on 5)

The method of claim 5, wherein the statistical difference measurement is one or more of: an arithmetic difference, a disparate impact ratio, a difference of means, and a Wasserstein distance.

### Claim 7 (depends on 5)

The method of claim 5, wherein the statistical difference measurement comprises a difference of model outputs, the model outputs being one or more of: log-odds scores, probabilities of classification, and class outputs.

### Claim 8

A non-transitory machine-readable medium storing instructions which, when executed by processing circuitry, cause the processing circuitry to:
receive a representation of a machine learning model, a representation of a first data segment, and a representation of a second data segment;
compute an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment;
determine a set of reasons for the computed output difference based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference; and
provide an output representing the set of reasons.

### Claim 9 (depends on 8)

The machine-readable medium of claim 8, wherein the instructions, when executed by the processing circuitry, further cause the processing circuitry to:
generate a remediation engine for the machine learning model to decrease a magnitude of the output difference, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features.

### Claim 10 (depends on 9)

The machine-readable medium of claim 9, wherein the remediation engine selects features for adjustment based on a user input received at the processing circuitry.

### Claim 11 (depends on 8)

The machine-readable medium of claim 8, wherein the set of metrics comprises a metric based on a Difference of Means or a Wasserstein Distance.

### Claim 12 (depends on 8)

The machine-readable medium of claim 8, wherein the output difference comprises a statistical difference measurement.

### Claim 13 (depends on 12)

The machine-readable medium of claim 12, wherein the statistical difference measurement is one or more of: an arithmetic difference, a disparate impact ratio, a difference of means, and a Wasserstein distance.

### Claim 14 (depends on 12)

The machine-readable medium of claim 12, wherein the statistical difference measurement comprises a difference of model outputs, the model outputs being one or more of: log-odds scores, probabilities of classification, and class outputs.

### Claim 15

A system comprising:
processing circuitry; and
a memory storing instructions which, when executed by the processing circuitry, cause the processing circuitry to:
receive a representation of a machine learning model, a representation of a first data segment, and a representation of a second data segment;
compute an output difference between an output of the machine learning model applied to the first data segment and an output of the machine learning model applied to the second data segment;
determine a set of reasons for the computed output difference based on a set of metrics defining distance between feature importance distributions, the set of reasons identifying a set of features from a feature vector of the machine learning model along with a relative contribution of each feature to the computed output difference; and
provide an output representing the set of reasons.

### Claim 16 (depends on 15)

The system of claim 15, wherein the instructions, when executed by the processing circuitry, further cause the processing circuitry to:
generate a remediation engine for the machine learning model to decrease a magnitude of the output difference, the remediation engine comprising a feature engineering sub-engine that adjusts features from the set of features.

### Claim 17 (depends on 16)

The system of claim 16, wherein the remediation engine selects features for adjustment based on a user input received at the processing circuitry.

### Claim 18 (depends on 15)

The system of claim 15, wherein the set of metrics comprises a metric based on a Difference of Means or a Wasserstein Distance.

### Claim 19 (depends on 15)

The machine-readable medium of claim 15, wherein the output difference comprises a statistical difference measurement.

### Claim 20 (depends on 19)

The machine-readable medium of claim 19, wherein the statistical difference measurement is one or more of: an arithmetic difference, a disparate impact ratio, a difference of means, and a Wasserstein distance.
