# US 2024/0256904 A1 · Semi-local model importance in feature space

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0256904 A1
- Document ID: US20240256904A1
- Status: Published application
- Kind: A1
- Application number: 18/425,822
- Filed: 2024-01-29
- Published: 2024-08-01
- Earliest claimed: 2023-01-30
- Assignees: The Toronto-Dominion Bank
- Applicants: The Toronto-Dominion Bank
- Inventors: Kin Kwan Leung; Saba Zuberi; Maksims Volkovs; Jianing Sun
- Web page: https://patentgrep.com/patent/US20240256904A1

## CPC

- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

To provide explanations for black box computer models, data samples are processed by the model to determine related feature attributions for each data sample, describing the extent to which feature values affect the model predictions for that data sample. A group of data samples is selected to be explained and the group is clustered into subgroups based on the feature attributions of the data samples. Because explanations related to feature attributions can be difficult to interpret or relate to input features, each of the subgroups is then described in the feature space, enabling ready interpretation of the groups at a semi-local level.

## Claims

### Claim 1

A system comprising:
a processor configured to execute instructions;
a non-transitory computer-readable medium containing instructions executable by the processor for:
generating a feature attribution with respect to an output of a computer model relative to input features for each data sample of a group of data samples;
clustering the group of data samples into a plurality of subgroups based on the respective feature attribution of each data sample; and
generating a feature region description in feature space with respect to input features for a subgroup of the plurality of subgroups.

### Claim 2 (depends on 1)

The system of claim 1, wherein the group of data samples is a subset of a set of data samples and the feature region description is determined with respect to the subgroup relative to the set of data samples.

### Claim 3 (depends on 1)

The system of claim 1, wherein the feature attribution is determined based on LIME, LRP, DeepLIFT, Integrated Gradients, Shapley Values, Grad-CAM, or Deep Taylor Decomposition.

### Claim 4 (depends on 1)

The system of claim 1, wherein the feature region description describes one or more rules with respect to one or more input features.

### Claim 5 (depends on 1)

The system of claim 1, wherein the feature region description is determined by training a decision tree with respect to membership in the subgroup.

### Claim 6 (depends on 1)

The system of claim 1, wherein the instructions are further executable for:
determining that a data sample is a member of a subgroup based on the feature region description;
identifying an action associated with the subgroup; and
performing the action for the data sample.

### Claim 7 (depends on 1)

The system of claim 1, wherein the instructions are further executable for providing a visualization for display to a user device, the visualization showing the feature region description relative to the group of data samples.

### Claim 8

A method, comprising:
generating a feature attribution with respect to an output of a computer model relative to input features for each data sample of a group of data samples;
clustering the group of data samples into a plurality of subgroups based on the respective feature attribution of each data sample; and
generating a feature region description in feature space with respect to input features for a subgroup of the plurality of subgroups.

### Claim 9 (depends on 8)

The method of claim 8, wherein the group of data samples is a subset of a set of data samples and the feature region description is determined with respect to the subgroup relative to the set of data samples.

### Claim 10 (depends on 8)

The method of claim 8, wherein the feature attribution is determined based on LIME, LRP, DeepLIFT, Integrated Gradients, Shapley Values, Grad-CAM, or Deep Taylor Decomposition.

### Claim 11 (depends on 8)

The method of claim 8, wherein the feature region description describes one or more rules with respect to one or more input features.

### Claim 12 (depends on 8)

The method of claim 8, wherein the feature region description is determined by training a decision tree with respect to membership in the subgroup.

### Claim 13 (depends on 8)

The method of claim 8, the method further comprising:
determining that a data sample is a member of a subgroup based on the feature region description;
identifying an action associated with the subgroup; and
performing the action for the data sample.

### Claim 14 (depends on 8)

The method of claim 8, the method further comprising providing a visualization for display to a user device, the visualization showing the feature region description relative to the group of data samples.

### Claim 15

A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to:
generate a feature attribution with respect to an output of a computer model relative to input features for each data sample of a group of data samples;
cluster the group of data samples into a plurality of subgroups based on the respective feature attribution of each data sample; and
generate a feature region description in feature space with respect to input features for a subgroup of the plurality of subgroups.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the group of data samples is a subset of a set of data samples and the feature region description is determined with respect to the subgroup relative to the set of data samples.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the feature attribution is determined based on LIME, LRP, DeepLIFT, Integrated Gradients, Shapley Values, Grad-CAM, or Deep Taylor Decomposition.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the feature region description describes one or more rules with respect to one or more input features.

### Claim 19 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the feature region description is determined by training a decision tree with respect to membership in the subgroup.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the instructions further cause the processor to:
determine that a data sample is a member of a subgroup based on the feature region description;
identify an action associated with the subgroup; and
perform the action for the data sample.
