# US 10,140,977 B1 · Generating additional training data for a natural language understanding engine > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,140,977 B1 - Document ID: US10140977B1 - Status: Granted - Kind: B1 - Application number: 16/051,362 - Filed: 2018-07-31 - Granted: 2018-11-27 - Earliest claimed: 2018-07-31 - Assignees: botbotbotbot Inc. - Applicants: botbotbotbot Inc. - Inventors: Antoine Raux; Yi Ma - Web page: https://patentgrep.com/patent/US10140977B1 ## CPC - G06N 20/00: Machine learning - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 40/20: Natural language analysis (semantic analysis of natural language G06F40/30) - G06F 40/35: Discourse or dialogue representation - G06F 40/51: Translation evaluation - G06N 3/02: Neural networks - G06N 3/044: Recurrent networks, e.g. Hopfield networks - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06N 3/091: Active learning - G10L 15/01: Assessment or evaluation of speech recognition systems - G10L 15/063: {Training} - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 2015/225: {Feedback of the input speech} - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating additional training data for a natural language understanding engine. One of the methods includes: obtaining data identifying (i) a first input conversational turn and (ii) a first annotation, determining that the first annotation accurately characterized the first input conversational turn, determining that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn, in response to the determining, obtaining one or more first paraphrases of the first input conversational turn; and generating, for each of the one or more first paraphrases, a respective first training example that identifies the first annotation as the correct annotation for the first paraphrase; and training the natural language understanding engine on at least the first training examples. ## Claims ### Claim 1 A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising: obtaining, during operation of a computer-implemented dialogue system comprising a natural language understanding engine, data identifying (i) a first input conversational turn that was provided as input to the natural language understanding engine during a dialogue between a user and the computer-implemented dialogue system and (ii) a first annotation of the first input conversational turn generated by the natural language understanding engine, wherein the natural language understanding engine has been trained on a first set of training data comprising a plurality of training conversational turns; determining that the first annotation accurately characterized the first input conversational turn; determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn; in response to determining that (i) the first annotation accurately characterized the first input conversational turn but (ii) the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn: obtaining one or more first paraphrases of the first input conversational turn; and generating, for each of the one or more first paraphrases, a respective first training example that identifies the first annotation as the correct annotation for the first paraphrase; and training the natural language understanding engine on at least the first training examples. ### Claim 2 (depends on 1) The system of claim 1, wherein determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn comprises: identifying training conversational turns in the training data that are similar to the first input conversational turn according to a similarity measure. ### Claim 3 (depends on 2) The system of claim 2, wherein identifying training conversational turns in the training data that are similar to the first input conversational turn according to a similarity measure comprises: determining a numeric embedding representation of the first input conversational turn; determining respective numeric embedding representations of each of the training conversational turns; and determining respective distances between the numeric embedding representation of the first input conversational turn and the numeric embedding representations of the training conversational turns. ### Claim 4 (depends on 2) The system of claim 2, wherein determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn comprises: determining that there are less than a threshold number of training conversational turns that are similar to the first input conversational turn according to the similarity measure. ### Claim 5 (depends on 2) The system of claim 2, wherein determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn comprises: determining that the natural language understanding unit performs poorly on training conversational turns that are similar to the first input conversational turn according to the similarity measure. ### Claim 6 (depends on 1) The system of claim 1, the operations further comprising obtaining a confidence score generated by the natural language understanding engine that represents a confidence that the first annotation is an accurate characterization of the first input conversational turn, and wherein determining that the first annotation accurately characterized the first input conversational turn comprises determining that the confidence score exceeds a threshold score. ### Claim 7 (depends on 1) The system of claim 1, wherein determining that the first annotation accurately characterized the first input conversational turn comprises: processing the first input conversational turn and the first annotation using a post-hoc annotation machine learning model that is configured to generate as output a quality score that represents a likelihood that the first annotation is an accurate characterization of the first input conversational turn; and determining that the quality score exceeds a threshold score. ### Claim 8 (depends on 7) The system of claim 7, wherein the post-hoc annotation machine learning model is configured to receive as input (i) the first input conversational turn, (ii) the first annotation, (iii) one or more conversational turns occurring before the first input conversational turn in the dialogue, and (iv) one or more conversational turns occurring after the first input conversational turn in the dialogue. ### Claim 9 (depends on 1) The system of claim 1, wherein determining that the first annotation accurately characterized the first input conversational turn comprises: determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation accurately characterizes the first input conversational turn. ### Claim 10 (depends on 9) The system of claim 9, wherein determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation accurately characterize the first input conversational turn comprises: determining (i) that a task was completed as a result of the dialogue and (ii) that no slot values assigned in the first annotation were changed or removed by the user in any of the conversational turns occurring after the first input conversational turn in the dialogue. ### Claim 11 (depends on 1) The system of claim 1, the operations further comprising: obtaining, during operation of the computer-implemented dialogue system, data identifying (i) a second input conversational turn that was provided as input to the natural language understanding engine during the dialogue between the user and the computer-implemented dialogue system and (ii) a second annotation of the second input conversational turn generated by the natural language understanding engine; determining that the second annotation did not accurately characterize the second input conversational turn; in response to determining that the second annotation did not accurately characterize the second input conversational turn: determining a correct annotation for the second conversational turn; obtaining one or more second paraphrases of the second input conversational turn; and generating, for each of the one or more second paraphrases, a respective second training example that identifies the correct annotation for the second conversational turn as the correct annotation for the second paraphrase; and training the natural language understanding engine on at least the second training examples. ### Claim 12 A method comprising: obtaining, during operation of a computer-implemented dialogue system comprising a natural language understanding engine, data identifying (i) a first input conversational turn that was provided as input to the natural language understanding engine during a dialogue between a user and the computer-implemented dialogue system and (ii) a first annotation of the first input conversational turn generated by the natural language understanding engine, wherein the natural language understanding engine has been trained on a first set of training data comprising a plurality of training conversational turns; determining that the first annotation accurately characterized the first input conversational turn; determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn; in response to determining that (i) the first annotation accurately characterized the first input conversational turn but (ii) the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn: obtaining one or more first paraphrases of the first input conversational turn; and generating, for each of the one or more first paraphrases, a respective first training example that identifies the first annotation as the correct annotation for the first paraphrase; and training the natural language understanding engine on at least the first training examples. ### Claim 13 (depends on 12) The method of claim 12, wherein determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn comprises: identifying training conversational turns in the training data that are similar to the first input conversational turn according to a similarity measure. ### Claim 14 (depends on 13) The method of claim 13, wherein identifying training conversational turns in the training data that are similar to the first input conversational turn according to a similarity measure comprises: determining a numeric embedding representation of the first input conversational turn; determining respective numeric embedding representations of each of the training conversational turns; and determining respective distances between the numeric embedding representation of the first input conversational turn and the numeric embedding representations of the training conversational turns. ### Claim 15 (depends on 13) The method of claim 13, wherein determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn comprises: determining that there are less than a threshold number of training conversational turns that are similar to the first input conversational turn according to the similarity measure. ### Claim 16 (depends on 13) The method of claim 13, wherein determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn comprises: determining that the natural language understanding unit performs poorly on training conversational turns that are similar to the first input conversational turn according to the similarity measure. ### Claim 17 (depends on 12) The method of claim 12, further comprising obtaining a confidence score generated by the natural language understanding engine that represents a confidence that the first annotation is an accurate characterization of the first input conversational turn, and wherein determining that the first annotation accurately characterized the first input conversational turn comprises determining that the confidence score exceeds a threshold score. ### Claim 18 (depends on 12) The method of claim 12, wherein determining that the first annotation accurately characterized the first input conversational turn comprises: processing the first input conversational turn and the first annotation using a post-hoc annotation machine learning model that is configured to generate as output a quality score that represents a likelihood that the first annotation is an accurate characterization of the first input conversational turn; and determining that the quality score exceeds a threshold score. ### Claim 19 (depends on 18) The method of claim 18, wherein the post-hoc annotation machine learning model is configured to receive as input (i) the first input conversational turn, (ii) the first annotation, (iii) one or more conversational turns occurring before the first input conversational turn in the dialogue, and (iv) one or more conversational turns occurring after the first input conversational turn in the dialogue. ### Claim 20 (depends on 12) The method of claim 12, wherein determining that the first annotation accurately characterized the first input conversational turn comprises: determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation accurately characterizes the first input conversational turn. ### Claim 21 (depends on 20) The method of claim 20, wherein determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation accurately characterize the first input conversational turn comprises: determining (i) that a task was completed as a result of the dialogue and (ii) that no slot values assigned in the first annotation were changed or removed by the user in any of the conversational turns occurring after the first input conversational turn in the dialogue. ### Claim 22 (depends on 12) The method of claim 12, further comprising: obtaining, during operation of the computer-implemented dialogue system, data identifying (i) a second input conversational turn that was provided as input to the natural language understanding engine during the dialogue between the user and the computer-implemented dialogue system and (ii) a second annotation of the second input conversational turn generated by the natural language understanding engine; determining that the second annotation did not accurately characterize the second input conversational turn; in response to determining that the second annotation did not accurately characterize the second input conversational turn: determining a correct annotation for the second conversational turn; obtaining one or more second paraphrases of the second input conversational turn; and generating, for each of the one or more second paraphrases, a respective second training example that identifies the correct annotation for the second conversational turn as the correct annotation for the second paraphrase; and training the natural language understanding engine on at least the second training examples. ### Claim 23 One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: obtaining, during operation of a computer-implemented dialogue system comprising a natural language understanding engine, data identifying (i) a first input conversational turn that was provided as input to the natural language understanding engine during a dialogue between a user and the computer-implemented dialogue system and (ii) a first annotation of the first input conversational turn generated by the natural language understanding engine, wherein the natural language understanding engine has been trained on a first set of training data comprising a plurality of training conversational turns; determining that the first annotation accurately characterized the first input conversational turn; determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn; in response to determining that (i) the first annotation accurately characterized the first input conversational turn but (ii) the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn: obtaining one or more first paraphrases of the first input conversational turn; and generating, for each of the one or more first paraphrases, a respective first training example that identifies the first annotation as the correct annotation for the first paraphrase; and training the natural language understanding engine on at least the first training examples. ### Claim 24 (depends on 23) The computer-readable storage media of claim 23, wherein determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn comprises: identifying training conversational turns in the training data that are similar to the first input conversational turn according to a similarity measure. ### Claim 25 (depends on 24) The computer-readable storage media of claim 24, wherein determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn comprises: determining that there are less than a threshold number of training conversational turns that are similar to the first input conversational turn according to the similarity measure. ### Claim 26 (depends on 24) The computer-readable storage media of claim 24, wherein determining, based on the training conversational turns in the first set of training data, that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn comprises: determining that the natural language understanding unit performs poorly on training conversational turns that are similar to the first input conversational turn according to the similarity measure. ### Claim 27 (depends on 23) The computer-readable storage media of claim 23, wherein determining that the first annotation accurately characterized the first input conversational turn comprises: processing the first input conversational turn and the first annotation using a post-hoc annotation machine learning model that is configured to generate as output a quality score that represents a likelihood that the first annotation is an accurate characterization of the first input conversational turn; and determining that the quality score exceeds a threshold score. ### Claim 28 (depends on 23) The computer-readable storage media of claim 23, wherein determining that the first annotation accurately characterized the first input conversational turn comprises: determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation accurately characterizes the first input conversational turn. ### Claim 29 (depends on 28) The computer-readable storage media of claim 28, wherein determining that conversational turns occurring after the first input conversational turn in the dialogue indicate that the first annotation accurately characterizes the first input conversational turn comprises: determining (i) that a task was completed as a result of the dialogue and (ii) that no slot values assigned in the first annotation were changed or removed by the user in any of the conversational turns occurring after the first input conversational turn in the dialogue. ### Claim 30 (depends on 23) The computer-readable storage media of claim 23, the operations further comprising: obtaining, during operation of the computer-implemented dialogue system, data identifying (i) a second input conversational turn that was provided as input to the natural language understanding engine during the dialogue between the user and the computer-implemented dialogue system and (ii) a second annotation of the second input conversational turn generated by the natural language understanding engine; determining that the second annotation did not accurately characterize the second input conversational turn; in response to determining that the second annotation did not accurately characterize the second input conversational turn: determining a correct annotation for the second conversational turn; obtaining one or more second paraphrases of the second input conversational turn; and generating, for each of the one or more second paraphrases, a respective second training example that identifies the correct annotation for the second conversational turn as the correct annotation for the second paraphrase; and training the natural language understanding engine on at least the second training examples.