US 11,636,387 B2Granted

System and method for improving machine learning models based on confusion error evaluation

AssigneeMicrosoft Technology Licensing, LLC

InventorsOren Elisha, Ami Luttwak, Hila Yehuda, Adar Kahana, Maya Bechler Speicher

Life of this patent

Estimating the term…

Loading patent…

Abstract

Embodiments described herein are directed to improving machine learning (ML) model-based techniques for automatically labeling data items based on identifying and resolving labels that are problematic. An ML model may be trained to predict labels for any given data item. The ML model may be validated to determine a confusion metric with respect to each distinct pair of labels predicted by the ML model. Each confusion metric indicates how a particular label is being mistaken for another particular label. The confusion metrics are analyzed to determine whether any of the ML model-generated labels are problematic (e.g., a label conflicts with another label, a label that is rarely predicted, a label that is incorrectly predicted, etc.). Steps for resolving the problematic labels are implemented, and the ML model is retrained based on the resolution steps. By doing so, the ML model generates a more accurate label for a data item.

Claims20

Dependent claims

A method, comprising:

receiving from a data store a plurality of data items and a user-generated label associated with each data item in the plurality of data items;
validating a machine learning (ML) model by:
training the ML model using a first portion of the plurality of data items and the user-generated labels associated therewith;
providing as input to the ML model a second portion of the plurality of data items;
receiving from the ML model one of a plurality of ML model-generated labels generated by the ML model for each data item in the second portion of the plurality of data items;
and
comparing the user-generated label associated with each data item in the second portion of the plurality of data items with the ML model-generated label associated with the same data item in the second portion of the plurality of data items;
based on the validating of the ML model, determining a confusion metric for each distinct pair of labels in the plurality of ML model-generated labels generated by the ML model, each confusion metric indicating a degree to which one label is being mistaken for the other label in the corresponding distinct pair of labels;
and
based on the confusion metrics, identifying one of the plurality of ML model-generated labels as a problematic label.

7 more elements

A system, comprising:

at least one processor circuit;
and
at least one memory that stores program code that, when executed by the at least one processor circuit, perform operations, the operations comprising:
receiving from a data store a plurality of data items and a user-generated label associated with each data item in the plurality of data items;
and
validating a machine learning (ML) model by:
training the ML model using a first portion of the plurality of data items and the user-generated labels associated therewith;
providing as input to the ML model a second portion of the plurality of data items;
receiving from the ML model one of a plurality of ML model-generated labels generated by the ML model for each data item in the second portion of the plurality of data items;
and
comparing the user-generated label associated with each data item in the second portion of the plurality of data items with the ML model-generated label associated with the same data item in the second portion of the plurality of data items;
based on the validating of the ML model, determining a confusion metric for each distinct pair of labels in the plurality of ML model-generated labels generated by the ML model, each confusion metric indicating a degree to which one label is being mistaken for the other label in the corresponding distinct pair of labels;
and
based on the confusion metrics, identifying one of the plurality of ML model-generated labels as a problematic label.

11 more elements

A computer-readable storage medium having program instructions recorded thereon that, when executed by at least one processor of a computing device, perform a method, the method comprising:

receiving from a data store a plurality of data items and a user-generated label associated with each data item in the plurality of data items;
validating a machine learning (ML) model by:
training the ML model using a first portion of the plurality of data items and the user-generated labels associated therewith;
providing as input to the ML model a second portion of the plurality of data items;
receiving from the ML model one of a plurality of ML model-generated labels generated by the ML model for each data item in the second portion of the plurality of data items;
and
comparing the user-generated label associated with each data item in the second portion of the plurality of data items with the ML model-generated label associated with the same data item in the second portion of the plurality of data items;
based on the validating of the ML model, determining a confusion metric for each distinct pair of labels in the plurality of ML model-generated labels generated by the ML model, each confusion metric indicating a degree to which one label is being mistaken for the other label in the corresponding distinct pair of labels;
and
based on the confusion metrics, identifying one of the plurality of ML model-generated labels as a problematic label.

7 more elements

Dependencies are inferred from the claim text.