US 12,755,208 B1 · System and method for generating high quality labeled extraction datasets for training machine learning models

Abstract

A labeled extraction dataset for training machine learning models for information extraction tasks is generated using an ensemble of machine learning models serving as annotators. The plurality of annotators generate a plurality of labels and associated confidences for each field in an input data source. The plurality of labels for each field are consolidated to generate a single label and associated confidence per field of the input data source. The quality of the resulting label is scored to identify high quality and low quality labels. The high quality labels are stored in a final labeled extraction dataset and the low quality labels are provided to human labels for review and correction. The final labeled extraction dataset may then be used in training machine learning models for information extraction tasks such as document understanding.

Claims20

Dependent claims

A method for generating a labeled extraction dataset for training machine learning models for information extraction tasks, comprising:

providing an input data source in an electronic format and a schema associated with the input data source to a plurality of annotators that are machine learning models and that each generate labels for the input data source based on the schema;
consolidating the labels generated for the input data source from the plurality of annotators into consolidated labels with a single label per field of the input data source;
scoring a quality of each consolidated label and identifying high quality consolidated labels and low quality consolidated labels based on a quality score for each consolidated label;
storing consolidated labels identified as high quality in a final labeled extraction dataset;
and
training a machine learning model for information extraction tasks using the final labeled extraction dataset.

3 more elements

A system for generating a labeled extraction dataset for training machine learning models for information extraction tasks, comprising:

one or more processors;
and
a memory communicatively coupled with the one or more processors and storing instructions that, when executed by the one or more processors, causes the one or more processors to be configured as:
provide an input data source in an electronic format and a schema associated with the input data source to a plurality of annotators that are machine learning models and that each generate labels for the input data source based on the schema;
consolidate the labels generated for the input data source from the plurality of annotators into consolidated labels with a single label per field of the input data source;
score a quality of each consolidated label and identifying high quality consolidated labels and low quality consolidated labels based on a quality score for each consolidated label;
store consolidated labels identified as high quality in a final labeled extraction dataset;
and
train a machine learning model for information extraction tasks using the final labeled extraction dataset.

6 more elements

Dependencies are inferred from the claim text.

System and method for generating high quality labeled extraction datasets for training machine learning models

Life of this patent

System and method for generating high quality labeled extraction datasets for training machine learning models

Life of this patent

Abstract

Claims20