US 12,603,085 B2Granted

Entity level data augmentation in chatbots for robust named entity recognition

AssigneeOracle International Corporation

InventorsSrinivasa Phani Kumar Gadde, Yuanxu Wu, Aashna Devang Kanuga, Elias Luqman Jalaluddin, Vishal Vishnoi, Mark Edward Johnson

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Techniques for data augmentation for training chatbot systems in natural language processing. In one particular aspect, a method is provided that includes generating a list of values to cover for an entity, selecting utterances from a set of data that have context for the entity, converting the utterances into templates, where each template of the templates comprises a slot that maps to the list of values for the entity, selecting a template from the templates, selecting a value from the list of values based on the mapping between the slot within the selected template and the list of values for the entity; and creating an artificial utterance based on the selected template and the selected value, where the creating the artificial utterance comprises inserting the selected value into the slot of the selected template that maps to the list of values for the entity.

Claims20

Dependent claims

A method comprising:

accessing an utterance received by a chatbot system comprising one or more chatbots;
using a trained machine learning model to extract entity information from the utterance, wherein the trained machine learning model is trained by:
training a machine learning model using a set of training data selected from a set of data comprising utterances, at least one utterance of the utterances comprising a given entity;
evaluating performance of the machine learning model with respect to the given entity using a set of testing data selected from the set of data;
in response to evaluating the performance of the machine learning model, determining that the set of training data should be augmented;
in response to determining that the set of training data should be augmented, generating an augmented set of training data by:
selecting a template from a plurality of templates associated with the given entity;
identifying a plurality of slots in the template, wherein the plurality of slots comprises a first slot that represents a value element for the given entity and a second slot that represents a function element for the given entity;
generating, based on an augmentation rate, a plurality of artificial utterances, wherein the plurality of artificial utterances is generated based on:
(i) inserting a value selected from a list of values for the given entity into the first slot;
and (ii) inserting a function output obtained by executing a function selected from a plurality of functions associated with the given entity into the second slot, wherein each value of the list of values representing a variation of the given entity, and wherein a number of the plurality of artificial utterances to be generated is determined based on the augmentation rate;
and
adding the plurality of artificial utterances to the set of training data;
and
using the augmented set of training data to further train the machine learning model to result in the trained machine learning model;
using the entity information identify a chatbot of the one or more chatbots;
and
routing the utterance to the chatbot.

15 more elements

A system comprising:

one or more processors;
and
a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:
accessing an utterance received by a chatbot system comprising one or more chatbots;
using a trained machine learning model to extract entity information from the utterance, wherein the trained machine learning model is trained by:
training a machine learning model using a set of training data selected from a set of data comprising utterances, at least one utterance of the utterances comprising a given entity;
evaluating performance of the machine learning model with respect to the given entity using a set of testing data selected from the set of data;
in response to evaluating the performance of the machine learning model, determining that the set of training data should be augmented;
in response to determining that the set of training data should be augmented, generating an augmented set of training data by:
selecting a template from a plurality of templates associated with the given entity;
identifying a plurality of slots in the template, wherein the plurality of slots comprises a first slot that represents a value element for the given entity and a second slot that represents a function element for the given entity;
generating, based on an augmentation rate, a plurality of artificial utterances, wherein the plurality of artificial utterances is generated based on:
(i) inserting a value selected from a list of values for the given entity into the first slot;
and (ii) inserting a function output obtained by executing a function selected from a plurality of functions associated with the given entity into the second slot, wherein each value of the list of values representing a variation of the given entity, and wherein a number of the plurality of artificial utterances to be generated is determined based on the augmentation rate;
and
adding the plurality of artificial utterances to the set of training data;
and
using the augmented set of training data to further train the machine learning model to result in the trained machine learning model;
using the entity information identify a chatbot of the one or more chatbots;
and
routing the utterance to the chatbot.

18 more elements

A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:

accessing an utterance received by a chatbot system comprising one or more chatbots;
using a trained machine learning model to extract entity information from the utterance, wherein the trained machine learning model is trained by:
training a machine learning model using a set of training data selected from a set of data comprising utterances, at least one utterance of the utterances comprising a given entity;
evaluating performance of the machine learning model with respect to the given entity using a set of testing data selected from the set of data;
in response to evaluating the performance of the machine learning model, determining that the set of training data should be augmented;
in response to determining that the set of training data should be augmented, generating an augmented set of training data by:
selecting a template from a plurality of templates associated with the given entity;
identifying a plurality of slots in the template, wherein the plurality of slots comprises a first slot that represents a value element for the given entity and a second slot that represents a function element for the given entity;
generating, based on an augmentation rate, a plurality of artificial utterances, wherein the plurality of artificial utterances is generated based on:
(i) inserting a value selected from a list of values for the given entity into the first slot;
and (ii) inserting a function output obtained by executing a function selected from a plurality of functions associated with the given entity into the second slot, wherein each value of the list of values representing a variation of the given entity, and wherein a number of the plurality of artificial utterances to be generated is determined based on the augmentation rate;
and
adding the plurality of artificial utterances to the set of training data;
and
using the augmented set of training data to further train the machine learning model to result in the trained machine learning model;
using the entity information identify a chatbot of the one or more chatbots;
and
routing the utterance to the chatbot.

15 more elements

Dependencies are inferred from the claim text.

Facts

Application: 18/471,491Filed 21 Sep 2023 · continuation
Priority: 12 Jun 2020
Published: US 2024/0013780 A111 Jan 2024
Granted: 14 Apr 2026
Term ends: Not estimatedThe source records don’t support an estimate.

Classification20 codes

G10L 15/197
Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules: probabilistic grammars, e.g. word n-grams
G06F 40/186
Editing, e.g. inserting or deleting: templates
G06F 40/295
Phrasal analysis, e.g. finite state techniques or chunking: named entity recognition
G06F 40/30
Handling natural language data: semantic analysis

Family

Application63/038,337Us provisional application · filed 12 Jun 2020
Application17/345,288Parent of this continuation · filed 11 Jun 2021
US 2024/0013780 A1Pre-grant publication · published 11 Jan 2024
US 11,804,219 B2Parent of this continuation
US 2021/0390951 A1Published application
US 12,603,085 B2This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key