System for automatically annotating training data for a natural language understanding system
AssigneeMicrosoft Corporation
InventorsAlejandro Acero, Ye-Yi Wang, Leon Wong
Life of this patent
Estimating the term…
Abstract
The present invention uses a natural language understanding system that is currently being trained to assist in annotating traomomg data for training that natural language understanding system. Unannotated training data is provided to the system and the system proposes annotations to the training data. The user is offered an opportunity to confirm or correct the proposed annotations, and the system is trained with the corrected or verified annotations.
Claims51
A method of generating annotated training data to train a natural language understanding (NLU) system having one or models, comprising:
- generating a proposed annotation with the NLU system for each unit of unannotated training data;
- displaying the proposed annotations for user verification or correction to obtain a user-confirmed annotation;
- and
- training the NLU system with the user-confirmed annotation.
A user interface for training a natural language understanding (NLU) system that has one or more models, the user interface comprising:
- a first portion displaying a model display representative of the one or more models;
- a second portion displaying unannotated training inputs;
- and
- a third portion displaying a proposed annotation for a selected one of the unannotated training inputs.
A method of generating annotated training data for training a natural language understanding (NLU) system having at least one model, comprising:
- generating a proposed annotation for a unit of unannotated training data;
- calculating a confidence measure for a plurality of different portions of the proposed annotation;
- and
- displaying the proposed annotation by visually contrasting portions that have a corresponding confidence measure that falls below a threshold level.
A method of generating annotated training data for training a natural language understanding (NLU) system having at least one model, comprising:
- generating, with the NLU system, a proposed annotation for a unit of unannotated training data;
- displaying the proposed annotation with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation;
- training the model with the user-confirmed annotation;
- and
- checking for inconsistencies among user-confirmed annotation and data already used to train the model by determining whether the model accurately predicts the prior user-confirmed annotations.
A method of generating annotated training data for training a natural language understanding (NLU) system, comprising:
- generating, with the NLU system, a proposed annotation for each of a plurality of units of unannotated training data;
- calculating a confidence measure for each of the proposed annotations;
- displaying the proposed annotations in an order based on the corresponding confidence measures, with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation.
A method of generating annotated training data for training a natural language understanding (NLU) system, comprising:
- generating, with the NLU system, a proposed annotation for each of a plurality of units of unannotated training data, each proposed annotation having a type;
- displaying the proposed annotations in an order based on the type, with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation.
A method of generating annotated training data for training a natural language understanding (NLU) system employing a plurality of different natural language understanding techniques, comprising:
- generating, with each natural language training technique, a proposed annotation for a unit of unannotated training data, to obtain a plurality of proposed annotations;
- selecting one of the plurality of proposed annotations;
- and
- displaying the selected proposed annotation, with user actuable inputs for user correction or verification of the proposed annotation to obtain a user-confirmed annotation.
Dependencies are inferred from the claim text.