# US 2021/0342707 A1 · Data-driven techniques for model ensembles

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342707 A1
- Document ID: US20210342707A1
- Status: Published application
- Kind: A1
- Application number: 15/929,428
- Filed: 2020-05-01
- Published: 2021-11-04
- Earliest claimed: 2020-05-01
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: JING XU; STEVEN GEORGE BARBEE; JI YANG; SI ER HAN; XUE YANG ZHANG
- Web page: https://patentgrep.com/patent/US20210342707A1

## CPC

- G06N 5/04: Inference or reasoning models
- G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
- G06N 20/20: Ensemble learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Techniques to ensemble machine learning (ML) models are provided. A plurality of residues is generated by processing a plurality of input records using a plurality of ML models. A plurality of data clusters is identified by evaluating, using a clustering model, the plurality of input records and the plurality of residues. A first ensemble is generated for a first data cluster of the plurality of data clusters, where the first ensemble comprises one or more of the plurality of ML models. Upon determining that a new input record corresponds to the first data cluster, the new input record is processed using the first ensemble.

## Claims

### Claim 1

A method, comprising:
generating a plurality of residues by processing a plurality of input records using a plurality of machine learning (ML) models;
identifying a plurality of data clusters by evaluating, using a clustering model, the plurality of input records and the plurality of residues;
generating a first ensemble for a first data cluster of the plurality of data clusters, wherein the first ensemble comprises one or more of the plurality of ML models; and
upon determining that a new input record corresponds to the first data cluster, processing the new input record using the first ensemble.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the plurality of residues comprises generating a set of residues for a first input record of the plurality of input records, comprising:
generating a first prediction by evaluating the first input record using a first ML model of the plurality of ML models;
determining a first residue by comparing the first prediction with a first label for the first input record;
generating a second prediction by evaluating the first input record using a second ML model of the plurality of ML models; and
determining a second residue by comparing the second prediction with the first label.

### Claim 3 (depends on 1)

The method of claim 1, wherein generating the first ensemble for the first data cluster comprises:
sorting the plurality of ML models based on their performance with respect to the first data cluster;
selecting a first ML model of the plurality of ML models, based on determining that the first ML model provides a highest performance of the plurality of ML models;
selecting a second ML model of the plurality of ML models, based on determining that the second ML model provides a second-highest performance of the plurality of ML models; and
generating the first ensemble to include the first and second ML models.

### Claim 4 (depends on 3)

The method of claim 3, wherein generating the first ensemble for the first data cluster further comprises:
evaluating the first ensemble; and
upon determining that performance of the first ensemble is below a predefined threshold:
selecting a third ML model of the plurality of ML models, based on determining that the third ML model provides a third-highest performance of the plurality of ML models; and
generating the first ensemble to include the first, second, and third ML models.

### Claim 5 (depends on 1)

The method of claim 1, the method further comprising:
evaluating the input records belonging to the first data cluster to generate an importance score of one or more data fields with respect to the first data cluster.

### Claim 6 (depends on 5)

The method of claim 5, wherein generating the importance score of one or more indicative fields comprises:
determining, for each of a plurality of data fields, a distribution of values in the plurality of input records; and
determining, for a first data field of the plurality of data fields, a distribution of values with respect to the first data cluster; and
generating an importance score for the first data field based on a difference between the distribution of values with respect to the first data cluster and the distribution of values in the plurality of input records.

### Claim 7 (depends on 1)

The method of claim 1, wherein determining that the new input record corresponds to the first data cluster comprises:
evaluating the new input record using the clustering model.

### Claim 8

One or more computer-readable storage media collectively containing computer program code that, when executed by operation of one or more computer processors, performs an operation comprising:
generating a plurality of residues by processing a plurality of input records using a plurality of machine learning (ML) models;
identifying a plurality of data clusters by evaluating, using a clustering model, the plurality of input records and the plurality of residues;
generating a first ensemble for a first data cluster of the plurality of data clusters, wherein the first ensemble comprises one or more of the plurality of ML models; and
upon determining that a new input record corresponds to the first data cluster, processing the new input record using the first ensemble.

### Claim 9 (depends on 8)

The computer-readable storage media of claim 8, wherein generating the plurality of residues comprises generating a set of residues for a first input record of the plurality of input records, comprising:
generating a first prediction by evaluating the first input record using a first ML model of the plurality of ML models;
determining a first residue by comparing the first prediction with a first label for the first input record;
generating a second prediction by evaluating the first input record using a second ML model of the plurality of ML models; and
determining a second residue by comparing the second prediction with the first label.

### Claim 10 (depends on 8)

The computer-readable storage media of claim 8, wherein generating the first ensemble for the first data cluster comprises:
sorting the plurality of ML models based on their performance with respect to the first data cluster;
selecting a first ML model of the plurality of ML models, based on determining that the first ML model provides a highest performance of the plurality of ML models;
selecting a second ML model of the plurality of ML models, based on determining that the second ML model provides a second-highest performance of the plurality of ML models; and
generating the first ensemble to include the first and second ML models.

### Claim 11 (depends on 10)

The computer-readable storage media of claim 10, wherein generating the first ensemble for the first data cluster further comprises:
evaluating the first ensemble; and
upon determining that performance of the first ensemble is below a predefined threshold:
selecting a third ML model of the plurality of ML models, based on determining that the third ML model provides a third-highest performance of the plurality of ML models; and
generating the first ensemble to include the first, second, and third ML models.

### Claim 12 (depends on 8)

The computer-readable storage media of claim 8, the operation further comprising:
evaluating the input records belonging to the first data cluster to generate an importance score of one or more data fields with respect to the first data cluster.

### Claim 13 (depends on 12)

The computer-readable storage media of claim 12, wherein generating the importance score of one or more indicative fields comprises:
determining, for each of a plurality of data fields, a distribution of values in the plurality of input records; and
determining, for a first data field of the plurality of data fields, a distribution of values with respect to the first data cluster; and
generating an importance score for the first data field based on a difference between the distribution of values with respect to the first data cluster and the distribution of values in the plurality of input records.

### Claim 14 (depends on 8)

The computer-readable storage media of claim 8, wherein determining that the new input record corresponds to the first data cluster comprises:
evaluating the new input record using the clustering model.

### Claim 15

A system comprising:
one or more computer processors; and
one or more memories collectively containing one or more programs which when executed by the one or more computer processors performs an operation, the operation comprising:
generating a plurality of residues by processing a plurality of input records using a plurality of machine learning (ML) models;
identifying a plurality of data clusters by evaluating, using a clustering model, the plurality of input records and the plurality of residues;
generating a first ensemble for a first data cluster of the plurality of data clusters, wherein the first ensemble comprises one or more of the plurality of ML models; and
upon determining that a new input record corresponds to the first data cluster, processing the new input record using the first ensemble.

### Claim 16 (depends on 15)

The system of claim 15, wherein generating the plurality of residues comprises generating a set of residues for a first input record of the plurality of input records, comprising:
generating a first prediction by evaluating the first input record using a first ML model of the plurality of ML models;
determining a first residue by comparing the first prediction with a first label for the first input record;
generating a second prediction by evaluating the first input record using a second ML model of the plurality of ML models; and
determining a second residue by comparing the second prediction with the first label.

### Claim 17 (depends on 15)

The system of claim 15, wherein generating the first ensemble for the first data cluster comprises:
sorting the plurality of ML models based on their performance with respect to the first data cluster;
selecting a first ML model of the plurality of ML models, based on determining that the first ML model provides a highest performance of the plurality of ML models;
selecting a second ML model of the plurality of ML models, based on determining that the second ML model provides a second-highest performance of the plurality of ML models; and
generating the first ensemble to include the first and second ML models.

### Claim 18 (depends on 17)

The system of claim 17, wherein generating the first ensemble for the first data cluster further comprises:
evaluating the first ensemble; and
upon determining that performance of the first ensemble is below a predefined threshold:
selecting a third ML model of the plurality of ML models, based on determining that the third ML model provides a third-highest performance of the plurality of ML models; and
generating the first ensemble to include the first, second, and third ML models.

### Claim 19 (depends on 15)

The system of claim 15, the operation further comprising:
evaluating the input records belonging to the first data cluster to generate an importance score of one or more data fields with respect to the first data cluster, wherein generating the importance score of one or more indicative fields comprises:
determining, for each of a plurality of data fields, a distribution of values in the plurality of input records; and
determining, for a first data field of the plurality of data fields, a distribution of values with respect to the first data cluster; and
generating an importance score for the first data field based on a difference between the distribution of values with respect to the first data cluster and the distribution of values in the plurality of input records.

### Claim 20 (depends on 15)

The system of claim 15, wherein determining that the new input record corresponds to the first data cluster comprises:
evaluating the new input record using the clustering model.
