# US 2021/0334693 A1 · Automated generation of explainable machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334693 A1
- Document ID: US20210334693A1
- Status: Published application
- Kind: A1
- Application number: 16/855,523
- Filed: 2020-04-22
- Published: 2021-10-28
- Earliest claimed: 2020-04-22
- Assignees: Intuit Inc.
- Applicants: Intuit Inc.
- Inventors: Nitzan BAVLY; Yehezkel Shraga RESHEFF; Tzvika BARENHOLZ; Talia TRON
- Web page: https://patentgrep.com/patent/US20210334693A1

## CPC

- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G06F 16/2365: {Ensuring data consistency and integrity}
- G06F 16/24578: {using ranking}
- G06N 5/04: Inference or reasoning models
- G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A computer-implemented method and system are provided to perform a machine learning pipeline process to produce an explainable machine learning model. A computing device may be configured to train a plurality of machine learning models with a set of respective feature datasets to generate an accuracy and explainability property for each trained model. The computing device may evaluate a plurality of the trained machine learning models and select a model as an explainable machine learning model based on at least one of the accuracy and the explainability property.

## Claims

### Claim 1

A method implemented by a computing system, the computing system comprising one or more processors and one or more non-transitory computer-readable storage devices having computer-executable computer instructions which, when executed by the one or more processors, cause the one or more processors to perform a machine learning pipeline process comprising:
Receiving feature datasets from a database;
obtaining a set of models, each model being selected to be compatible with at least one of the feature datasets;
training each model of the set of models with the at least one feature dataset to create a set of trained models;
generating an accuracy value and explainability properties for each model of the set of the trained models; and
selecting an explainable model as a recommended model from the set of the trained models based on the accuracy value and the explainability properties.

### Claim 2 (depends on 1)

The method of claim 1, wherein the machine learning pipeline process further comprises:
ranking the set of the trained models based on the accuracy value and the explainability properties of each model in the set of trained models; and
utilizing the explainability properties of the ranked set of trained models to define a subset of the trained models from which to select the explainable model.

### Claim 3 (depends on 2)

The method of claim 2, wherein said utilizing step further comprises:
determining, from the ranked set of trained models, the subset of trained models by selecting trained models having respective explainability properties above a predetermined explainability threshold; and
determining, from the subset of the trained models, the explainable model as the trained model with a maximum accuracy value.

### Claim 4 (depends on 1)

The method of claim 1, wherein the machine learning pipeline process further comprises:
ranking the set of trained models based on the accuracy value and the explainability properties of each model in the set of trained models; and
utilizing the accuracy value of the ranked set of trained models to define a subset of the trained models from which to select the explainable model.

### Claim 5 (depends on 4)

The method of claim 4, wherein said utilizing step further comprises:
determining, from the ranked set of trained models, the subset of trained models by selecting trained models having an accuracy value above a predetermined accuracy threshold; and
determining, from the subset of the trained models, the explainable model as the trained model with the best explainability properties.

### Claim 6 (depends on 1)

The method of claim 1, wherein the accuracy value of each trained model is measured based on a cross-validation procedure and a performance score of each trained model and the explainability properties of each trained model are generated by a hard-coded ranking of the trained models.

### Claim 7 (depends on 1)

The method of claim 1, wherein each feature comprises a flag to indicate whether the feature has a semantic meaning that may be used as an explanation for a trained model.

### Claim 8

A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable storage devices storing computer-executable instructions, the instructions operable to cause the one or more processors to perform a machine learning pipeline process comprising:
receiving feature datasets from a database;
obtaining a set of models, each model being selected to be compatible with at least one of the feature datasets;
training each model of the set of models with the at least one feature dataset to create a set of trained models;
generating an accuracy value and explainability properties for each model of the set of the trained models; and
selecting an explainable model as a recommended model from the set of the trained models based on the accuracy value and the explainability properties.

### Claim 9 (depends on 8)

The system of claim 8, wherein the machine learning pipeline process further comprises:
ranking the set of the trained models based on the accuracy value and the explainability properties of each model in the set of trained models; and
utilizing the explainability properties of the ranked set of trained models to define a subset of the trained models from which to select the explainable model.

### Claim 10 (depends on 9)

The system of claim 9, wherein said utilizing step further comprises:
determining, from the ranked set of trained models, the subset of trained models by selecting trained models having respective explainability properties above a predetermined explainability threshold; and
determining, from the subset of the trained models, the explainable model as the trained model with a maximum accuracy value.

### Claim 11 (depends on 8)

The system of claim 8, wherein the machine learning pipeline process further comprises:
ranking the set of trained models based on the accuracy value and the explainability properties of each model in the set of trained models; and
utilizing the accuracy value of the ranked set of trained models to define a subset of the trained models from which to select the explainable model.

### Claim 12 (depends on 11)

The system of claim 11, wherein said utilizing step further comprises:
determining, from the ranked set of trained models, the subset of trained models by selecting trained models having an accuracy value above a predetermined accuracy threshold; and
determining, from the subset of the trained models, the explainable model as the trained model with the best explainability properties.

### Claim 13 (depends on 8)

The system of claim 8, wherein the accuracy of each trained model is measured based on a cross-validation procedure and a performance score of each trained model; and the explainability property of each trained model is generated by a hard-coded ranking of the trained models.

### Claim 14 (depends on 10)

The system of claim 10, wherein each feature comprises a flag to indicate whether the feature has a semantic meaning that may be used as an explanation for a trained model.

### Claim 15

A computing system, comprising:
one or more processors; and
one or more non-transitory computer-readable storage devices storing computer-executable instructions, the instructions operable to cause the one or more processors to perform a machine learning pipeline process comprising:
receiving feature datasets from a database;
obtaining a set of models, each model being selected to be compatible with at least one of the feature datasets;
training each model of the set of models with the at least one feature dataset to create a set of trained models;
generating an accuracy value and explainability properties for each model of the set of the trained models;
ranking the set of the trained models based on the accuracy value and the explainability properties of each model in the set of trained models; and
selecting an explainable model as a recommended model from the set of the trained models based on the accuracy value and the explainability properties of the ranked set of trained models.

### Claim 16 (depends on 15)

The system of claim 15, wherein the machine learning pipeline process further comprises:
utilizing the explainability properties of the ranked set of trained models to define a subset of the trained models from which to select the explainable model.

### Claim 17 (depends on 16)

The system of claim 16, wherein said utilizing step further comprises:
determining, from the ranked set of trained models, the subset of trained models by selecting trained models having respective explainability properties above a predetermined explainability threshold; and
determining, from the subset of the trained models, the explainable model as the trained model with a maximum accuracy value.

### Claim 18 (depends on 15)

The system of claim 15, wherein the machine learning pipeline process further comprises:
utilizing the accuracy value of the ranked set of trained models to define a subset of the trained models from which to select the explainable model.

### Claim 19 (depends on 18)

The system of claim 18, wherein said utilizing step further comprises:
determining, from the ranked set of trained models, the subset of trained models by selecting trained models having an accuracy value above a predetermined accuracy threshold; and
determining, from the subset of the trained models, the explainable model as the trained model with the best explainability properties.

### Claim 20 (depends on 15)

The system of claim 15, wherein each feature comprises a flag to indicate whether the feature has a semantic meaning that may be used as an explanation for a trained model.
