US 10,140,977 B1Granted

Generating additional training data for a natural language understanding engine

Assigneebotbotbotbot Inc.

InventorsAntoine Raux, Yi Ma

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating additional training data for a natural language understanding engine. One of the methods includes: obtaining data identifying (i) a first input conversational turn and (ii) a first annotation, determining that the first annotation accurately characterized the first input conversational turn, determining that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn, in response to the determining, obtaining one or more first paraphrases of the first input conversational turn; and generating, for each of the one or more first paraphrases, a respective first training example that identifies the first annotation as the correct annotation for the first paraphrase; and training the natural language understanding engine on at least the first training examples.

Claims30

Dependent claims

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

obtaining, during operation of a computer-implemented dialogue system comprising a natural language understanding engine, data identifying (i) a first input conversational turn that was provided as input to the natural language understanding engine during a dialogue between a user and the computer-implemented dialogue system and (ii) a first annotation of the first input conversational turn generated by the natural language understanding engine, wherein the natural language understanding engine has been trained on a first set of training data comprising a plurality of training conversational turns;
determining that the first annotation accurately characterized the first input conversational turn;
determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn;
in response to determining that (i) the first annotation accurately characterized the first input conversational turn but (ii) the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn:
obtaining one or more first paraphrases of the first input conversational turn;
and
generating, for each of the one or more first paraphrases, a respective first training example that identifies the first annotation as the correct annotation for the first paraphrase;
and
training the natural language understanding engine on at least the first training examples.

6 more elements

A method comprising:

obtaining, during operation of a computer-implemented dialogue system comprising a natural language understanding engine, data identifying (i) a first input conversational turn that was provided as input to the natural language understanding engine during a dialogue between a user and the computer-implemented dialogue system and (ii) a first annotation of the first input conversational turn generated by the natural language understanding engine, wherein the natural language understanding engine has been trained on a first set of training data comprising a plurality of training conversational turns;
determining that the first annotation accurately characterized the first input conversational turn;
determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn;
in response to determining that (i) the first annotation accurately characterized the first input conversational turn but (ii) the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn:
obtaining one or more first paraphrases of the first input conversational turn;
and
generating, for each of the one or more first paraphrases, a respective first training example that identifies the first annotation as the correct annotation for the first paraphrase;
and
training the natural language understanding engine on at least the first training examples.

6 more elements

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining, during operation of a computer-implemented dialogue system comprising a natural language understanding engine, data identifying (i) a first input conversational turn that was provided as input to the natural language understanding engine during a dialogue between a user and the computer-implemented dialogue system and (ii) a first annotation of the first input conversational turn generated by the natural language understanding engine, wherein the natural language understanding engine has been trained on a first set of training data comprising a plurality of training conversational turns;
determining that the first annotation accurately characterized the first input conversational turn;
determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn;
in response to determining that (i) the first annotation accurately characterized the first input conversational turn but (ii) the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn:
obtaining one or more first paraphrases of the first input conversational turn;
and
generating, for each of the one or more first paraphrases, a respective first training example that identifies the first annotation as the correct annotation for the first paraphrase;
and
training the natural language understanding engine on at least the first training examples.

6 more elements

Dependencies are inferred from the claim text.