# US 2019/0378210 A1 · Systems and methods for decomposition of non-differentiable and differentiable models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0378210 A1
- Document ID: US20190378210A1
- Status: Published application
- Kind: A1
- Application number: 16/434,731
- Filed: 2019-06-07
- Published: 2019-12-12
- Earliest claimed: 2018-06-08
- Assignees: ZestFinance, Inc.
- Applicants: ZestFinance, Inc.
- Inventors: Douglas C. Merrill; Michael Edward Ruberry; Sean Javad Kamkar; Jerome Louis Budzik; John Wickens Lamb Merrill
- Web page: https://patentgrep.com/patent/US20190378210A1

## CPC

- G06N 20/20: Ensemble learning
- G06Q 40/03: Credit; Loans; Processing thereof
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 5/02: Knowledge representation; Symbolic representation
- G06N 20/00: Machine learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/048: Activation functions
- G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Systems and methods for explaining non-differentiable models and differentiable models.

## Claims

### Claim 1

A method comprising: with a model explanation system:
accessing non-differentiable model information of a non-differentiable model;
selecting a reference population of one or more reference data points;
selecting a test population of one or more test data points;
for each test data point, generating a non-differentiable decomposition value for each feature of the test data point, wherein generating a non-differentiable decomposition value comprises:
generating a SHAP (SHapley Additive exPlanation) value for the corresponding feature for the corresponding test data point by using the non-differentiable model information;
generating a SHAP value for the corresponding feature for each reference data point by using the non-differentiable model information;
for each reference data point, generating a difference value between the SHAP value for the test data point and the SHAP value of the reference data point;
generating the non-differentiable decomposition value by averaging the generated difference values;
generating explanation information based on at least the generated non-differentiable decomposition values;
providing the generated explanation information to an external system.

### Claim 2 (depends on 1)

The method of claim 1, wherein the external system includes one or more of an operator device and a modeling system.

### Claim 3 (depends on 1)

The method of claim 1, wherein the non-differentiable model is a tree-based model.

### Claim 4 (depends on 1)

The method of claim 1, wherein generating explanation information based on at least the generated non-differentiable decomposition values comprises: for each test data point, accessing a model result generated for the test data point by using the non-differentiable model, and mapping the model result to the non-differentiable decomposition values generated for the test data point, wherein the explanation information is generated by using the mappings of model results to non-differentiable decomposition values.

### Claim 5 (depends on 3)

The method of claim 3,
wherein the non-differentiable model is a credit risk model,
wherein selecting a reference population comprises selecting input data sets of applicants approved by the credit risk model,
wherein selecting a test population comprises selecting an input data set of an applicant denied by the credit risk model,
wherein generating explanation information based on at least the generated non-differentiable decomposition values comprises:
accessing a score generated for the test data point by using the credit risk model,
accessing adverse action reason codes from at least one of a storage device and a modeling system,
selecting at least one of the adverse action reason codes based on the non-differentiable decomposition values generated for the input data set of the applicant denied by the credit risk model,
generating the explanation information, wherein the explanation information identifies the selected adverse action reason codes and the accessed score.

### Claim 6 (depends on 3)

The method of claim 3,
wherein the non-differentiable model is a credit risk model,
wherein selecting a test population of one or more test data points comprises selecting input data sets of applicants that are members of a protected class,
wherein selecting a reference population comprises selecting input data sets of applicants that are members of a reference population to be compared with the members of the protected class,
wherein generating explanation information based on at least the generated non-differentiable decomposition values comprises:
for each feature, averaging the non-differentiable decomposition values generated for each test data point to generate a protected class non-differentiable decomposition value,
selecting features having a protected class non-differentiable decomposition value above a threshold,
generating the explanation information, wherein the explanation information identifies the selected features.

### Claim 7 (depends on 6)

The method of claim 6, wherein generating explanation information based on at least the generated non-differentiable decomposition values further comprises:
determining whether any of the selected features are impermissible for use in the credit risk model, and
identifying any features determined to be impermissible in the generated explanation information.

### Claim 8 (depends on 7)

The method of claim 7, wherein determining whether any of the selected features are impermissible for use in the credit risk model comprises:
accessing a predetermined list of impermissible features.

### Claim 9 (depends on 7)

The method of claim 7, wherein determining whether any of the selected features are impermissible for use in the credit risk model comprises:
accessing training data for the credit risk model,
accessing an original credit score generated for each test data point by the credit risk model,
for each selected feature:
removing the feature from the accessed training data to generate comparison training data,
training the credit risk model by using the comparison training data to generate a comparison credit risk model for the feature,
generating a comparison credit score for each test data point by using the comparison credit risk model,
for each test data point, comparing the comparison credit score and the original credit score,
identifying the feature as impermissible based on the comparing of the comparison credit score and the original credit score for each test data point.

### Claim 10 (depends on 9)

The method of claim 9,
wherein comparing the comparison credit score and the original credit score comprises determining a difference between the comparison credit score and the original credit score,
wherein identifying the feature as impermissible based on the comparing of the comparison credit score and the original credit score for each test data point comprises:
determining an average of the differences determined for each test data point, and
identifying the feature as impermissible if the average of the differences exceeds a permissibility threshold.

### Claim 11 (depends on 7)

The method of claim 7, wherein selecting a reference population comprises selecting input data sets of White Non-Hispanic credit applicants.

### Claim 12 (depends on 3)

The method of claim 3, further comprising,
with the model explanation system, storing in a knowledge graph: each generated non-differentiable decomposition value, information identifying the test population and the reference population, the non-differentiable model information, and the generated explanation information; and
automatically generating model risk management documentation from the information stored in the knowledge graph.

### Claim 13 (depends on 3)

The method of claim 3,
wherein the non-differentiable model is a credit risk model,
wherein selecting a reference population comprises selecting input data sets of a first time period,
wherein selecting a test population of one or more test data points comprises selecting input data sets of a second time period,
wherein generating explanation information based on at least the generated non-differentiable decomposition values comprises:
for each feature, averaging the non-differentiable decomposition values generated for each test data point to generate a second time period non-differentiable decomposition value,
selecting features having a second time period non-differentiable decomposition value above a threshold,
generating the explanation information, wherein the explanation information identifies the selected features.

### Claim 14

A method comprising: with a model explanation system
accessing non-differentiable model information of a non-differentiable model of an ensemble and accessing differentiable model information of a differentiable model of the ensemble;
selecting a reference population of one or more reference data points;
selecting a test population of one or more test data points;
for each test data point, generating a non-differentiable decomposition value for each feature of the test data point, wherein generating a non-differentiable decomposition value comprises:
generating a SHAP (SHapley Additive exPlanation) value for the corresponding feature for the corresponding test data point by using the non-differentiable model information;
generating a SHAP value for the corresponding feature for each reference data point by using the non-differentiable model information;
for each reference data point, generating a difference value between the SHAP value for the test data point and the SHAP value of the reference data point;
generating the non-differentiable decomposition value by averaging the generated difference values;
for each test data point, generating a differentiable decomposition value for each feature of the test data point, wherein generating a differentiable decomposition value comprises:
for each reference data point, performing an integrated gradients process using the test data point and the reference data point to generate an integrated gradient value by using the differentiable model information of the differentiable model; and
generating the differentiable decomposition value by averaging the generated integrated gradients values
for each test data point, generating ensemble decomposition values, comprising: combining non-differentiable decomposition values with corresponding differentiable decomposition values by using an ensembling function of the ensemble;
generating explanation information based on at least the generated ensemble decomposition values; and
providing the generated explanation information to an external system.

### Claim 15 (depends on 14)

The method of claim 14, wherein the non-differentiable model is a tree model and the differentiable model is a neural network.

### Claim 16 (depends on 15)

The method of claim 15, wherein the ensemble is a continuous function.

### Claim 17 (depends on 16)

The method of claim 16,
wherein the ensemble is a credit risk model,
wherein selecting a reference population comprises selecting input data sets of applicants approved by the credit risk model,
wherein selecting a test population comprises selecting an input data set of an applicant denied by the credit risk model,
wherein generating explanation information based on at least the generated ensemble decomposition values comprises:
accessing a score generated for the test data point by using the credit risk model,
accessing adverse action reason codes from at least one of a storage device and a modeling system,
selecting at least one of the adverse action reason codes based on the ensemble decomposition values generated for the input data set of the applicant denied by the credit risk model,
generating the explanation information, wherein the explanation information identifies the selected adverse action reason codes and the accessed score.

### Claim 18 (depends on 16)

The method of claim 16,
wherein the ensemble is a credit risk model,
wherein selecting a test population of one or more test data points comprises selecting input data sets of applicants that are members of a protected class,
wherein selecting a reference population comprises selecting input data sets of applicants that are members of a reference population to be compared with the members of the protected class,
wherein generating explanation information based on at least the generated ensemble decomposition values comprises:
for each feature, averaging the ensemble decomposition values generated for each test data point to generate a protected class ensemble decomposition value,
selecting features having a protected class ensemble decomposition value above a threshold,
generating the explanation information, wherein the explanation information identifies the selected features.

### Claim 19 (depends on 16)

The method of claim 16,
wherein the ensemble model is a credit risk model,
wherein selecting a reference population comprises selecting input data sets of a first time period,
wherein selecting a test population of one or more test data points comprises selecting input data sets of a second time period,
wherein generating explanation information based on at least the generated ensemble decomposition values comprises:
for each feature, averaging the ensemble decomposition values generated for each test data point to generate a second time period ensemble decomposition value,
selecting features having a second time period ensemble decomposition value above a threshold,
generating the explanation information, wherein the explanation information identifies the selected features.

### Claim 20 (depends on 16)

The method of claim 16, further comprising, with the model explanation system:
storing in a knowledge graph: each generated non-differentiable decomposition value, each generated differentiable decomposition value, each generated ensemble decomposition value, information identifying the test population and the reference population, the non-differentiable model information, the differentiable model information, and the generated explanation information; and
automatically generating model risk management artifacts and documentation from the information stored in the knowledge graph.
