# US 2021/0232966 A1 · System and method for improving machine learning models based on confusion error evaluation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0232966 A1
- Document ID: US20210232966A1
- Status: Published application
- Kind: A1
- Application number: 16/773,153
- Filed: 2020-01-27
- Published: 2021-07-29
- Earliest claimed: 2020-01-27
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Oren Elisha; Ami Luttwak; Hila Yehuda; Adar Kahana; Maya Bechler Speicher
- Web page: https://patentgrep.com/patent/US20210232966A1

## CPC

- G06F 16/906: Clustering; Classification
- G06N 20/00: Machine learning
- G06N 5/04: Inference or reasoning models
- G06Q 10/10: Office automation; Time management
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR

## Abstract

Embodiments described herein are directed to improving machine learning (ML) model-based techniques for automatically labeling data items based on identifying and resolving labels that are problematic. An ML model may be trained to predict labels for any given data item. The ML model may be validated to determine a confusion metric with respect to each distinct pair of labels predicted by the ML model. Each confusion metric indicates how a particular label is being mistaken for another particular label. The confusion metrics are analyzed to determine whether any of the ML model-generated labels are problematic (e.g., a label conflicts with another label, a label that is rarely predicted, a label that is incorrectly predicted, etc.). Steps for resolving the problematic labels are implemented, and the ML model is retrained based on the resolution steps. By doing so, the ML model generates a more accurate label for a data item.

## Claims

### Claim 1

A method, comprising:
receiving from a data store a plurality of data items and a user-generated label associated with each data item in the plurality of data items;
validating a machine learning (ML) model by:
training the ML model using a first portion of the plurality of data items and the user-generated labels associated therewith;
providing as input to the ML model a second portion of the plurality of data items;
receiving from the ML model one of a plurality of ML model-generated labels for each data item in the second portion of the plurality of data items; and
comparing the user-generated label associated with each data item in the second portion of the plurality of data items with the ML model-generated label associated with the same data item in the second portion of the plurality of data items;
based on the validating of the ML model, determining a confusion metric for each distinct pair of labels in the plurality of ML model-generated labels; and
based on the confusion metrics, identifying one of the plurality of ML model-generated labels as a problematic label.

### Claim 2 (depends on 1)

The method of claim 1, wherein the plurality of data items comprises at least one of:
emails;
customer relationship management application-related messages; or
support ticketing system-related messages.

### Claim 3 (depends on 1)

The method of claim 1, wherein identifying the one of the plurality of ML model-generated labels as problematic comprises:
identifying the one of the plurality of ML model-generated labels as being in conflict with another label of the plurality of ML model-generated labels.

### Claim 4 (depends on 3)

The method of claim 3, further comprising performing, responsive to the identifying the one of the plurality of ML model-generated labels as problematic:
merging the problematic label with the other label that is conflict with the problematic label; and
retraining the ML model after said merging.

### Claim 5 (depends on 1)

The method of claim 1, wherein identifying the one of the plurality of ML model-generated labels as problematic comprises:
identifying the one of the plurality of ML model-generated labels as a relatively weak label such that other labels of the plurality of ML model-generated labels are predicted by the ML model instead of the one of the plurality of ML model-generated labels.

### Claim 6 (depends on 5)

The method of claim 5, further comprising performing, responsive to the identifying the one of the plurality of ML model-generated labels as problematic:
removing the problematic label; and
retraining the ML model after said removing.

### Claim 7 (depends on 1)

The method of claim 1, wherein identifying the one of the plurality of ML model-generated labels as problematic comprises:
identifying the one of the plurality of ML model-generated labels as disturbing other labels of the plurality of ML model-generated labels such that the one of the plurality of ML model-generated labels is incorrectly predicted by the ML model instead of the other labels.

### Claim 8 (depends on 7)

The method of claim 7, further comprising performing, responsive to the identifying the one of the plurality of ML model-generated labels as problematic:
providing additional data items associated with the problematic label to the ML model; and
retraining the ML model based on said providing.

### Claim 9 (depends on 1)

The method of claim 1, wherein the validating is performed a k number of iterations, wherein k is any positive integer, and wherein the confusion metric for each distinct pair of labels in the plurality of ML model-generated labels is aggregated across the k iterations.

### Claim 10

A system, comprising:
at least one processor circuit; and
at least one memory that stores program code configured to be executed by the at least one processor circuit, the program code comprising:
a data fetcher configured to receive from a data store a plurality of data items and a user-generated label associated with each data item in the plurality of data items; and
an artificial intelligence engine configured to:
validate a machine learning (ML) model by:
training the ML model using a first portion of the plurality of data items and the user-generated labels associated therewith;
providing as input to the ML model a second portion of the plurality of data items;
receiving from the ML model one of a plurality of ML model-generated labels for each data item in the second portion of the plurality of data items; and
comparing the user-generated label associated with each data item in the second portion of the plurality of data items with the ML model-generated label associated with the same data item in the second portion of the plurality of data items;
based on the validating of the ML model, determine a confusion metric for each distinct pair of labels in the plurality of ML model-generated labels; and
based on the confusion metrics, identify one of the plurality of ML model-generated labels as a problematic label.

### Claim 11 (depends on 10)

The system of claim 10, wherein the plurality of data items comprises at least one of:
e-mails;
customer relationship management application-related messages; or
support ticketing system-related messages.

### Claim 12 (depends on 10)

The system of claim 10, wherein the artificial intelligence engine is configured to identify the one of the plurality of ML model-generated labels as problematic by:
identifying the one of the plurality of ML model-generated labels as being in conflict with another label of the plurality of ML model-generated labels.

### Claim 13 (depends on 12)

The system of claim 12, wherein the artificial intelligence engine is further configured to perform, responsive to identifying the one of the plurality of ML model-generated labels as problematic,
merging the problematic label with the other label that is conflict with the problematic label; and
retraining the ML model after said merging.

### Claim 14 (depends on 10)

The system of claim 10, wherein the artificial intelligence engine is configured to identify the one of the plurality of ML model-generated labels as problematic by:
identifying the one of the plurality of ML model-generated labels as a relatively weak label such that other labels of the plurality of ML model-generated labels are predicted by the ML model instead of the one of the plurality of ML model-generated labels.

### Claim 15 (depends on 14)

The system of claim 14, wherein the artificial intelligence engine is further configured to perform, responsive to identifying the one of the plurality of ML model-generated labels as problematic,
removing the problematic label; and
retraining the ML model after said removing.

### Claim 16 (depends on 10)

The system of claim 10, wherein the artificial intelligence engine is configured to identify the one of the plurality of ML model-generated labels as problematic by:
identifying the one of the plurality of ML model-generated labels as disturbing other labels of the plurality of ML model-generated labels such that the one of the plurality of ML model-generated labels is incorrectly predicted by the ML model instead of the other labels.

### Claim 17 (depends on 16)

The system of claim 16, wherein the artificial intelligence engine is further configured to perform, responsive to identifying the one of the plurality of ML model-generated labels as problematic,
providing additional data items associated with the problematic label to the ML model; and
retraining the ML model based on said providing.

### Claim 18

A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processor of a computing device, perform a method, the method comprising:
receiving from a data store a plurality of data items and a user-generated label associated with each data item in the plurality of data items;
validating a machine learning (ML) model by:
training the ML model using a first portion of the plurality of data items and the user-generated labels associated therewith;
providing as input to the ML model a second portion of the plurality of data items;
receiving from the ML model one of a plurality of ML model-generated labels for each data item in the second portion of the plurality of data items; and
comparing the user-generated label associated with each data item in the second portion of the plurality of data items with the ML model-generated label associated with the same data item in the second portion of the plurality of data items;
based on the validating of the ML model, determining a confusion metric for each distinct pair of labels in the plurality of ML model-generated labels; and
based on the confusion metrics, identifying one of the plurality of ML model-generated labels as a problematic label.

### Claim 19 (depends on 18)

The computer-readable storage medium of claim 18, wherein the plurality of data items comprises at least one of:
e-mails;
customer relationship management application-related messages; or
support ticketing system-related messages.

### Claim 20 (depends on 18)

The computer-readable storage medium of claim 18, wherein identifying the one of the plurality of ML model-generated labels as problematic comprises:
identifying the one of the plurality of ML model-generated labels as being in conflict with another label of the plurality of ML model-generated labels.
