# US 12,603,085 B2 · Entity level data augmentation in chatbots for robust named entity recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,603,085 B2
- Document ID: US12603085B2
- Status: Granted
- Kind: B2
- Application number: 18/471,491
- Filed: 2023-09-21
- Published: 2024-01-11
- Granted: 2026-04-14
- Earliest claimed: 2020-06-12
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Srinivasa Phani Kumar Gadde; Yuanxu Wu; Aashna Devang Kanuga; Elias Luqman Jalaluddin; Vishal Vishnoi; Mark Edward Johnson
- Web page: https://patentgrep.com/patent/US12603085B2

## CPC

- G10L 15/197: Probabilistic grammars, e.g. word n-grams
- G06F 40/186: Templates
- G06F 40/295: Named entity recognition
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G10L 15/063: {Training}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
- H04L 51/52: for supporting social networking services
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G10L 2015/0631: {Creating reference templates; Clustering}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Techniques for data augmentation for training chatbot systems in natural language processing. In one particular aspect, a method is provided that includes generating a list of values to cover for an entity, selecting utterances from a set of data that have context for the entity, converting the utterances into templates, where each template of the templates comprises a slot that maps to the list of values for the entity, selecting a template from the templates, selecting a value from the list of values based on the mapping between the slot within the selected template and the list of values for the entity; and creating an artificial utterance based on the selected template and the selected value, where the creating the artificial utterance comprises inserting the selected value into the slot of the selected template that maps to the list of values for the entity.

## Claims

### Claim 1

A method comprising:
accessing an utterance received by a chatbot system comprising one or more chatbots;
using a trained machine learning model to extract entity information from the utterance, wherein the trained machine learning model is trained by:
training a machine learning model using a set of training data selected from a set of data comprising utterances, at least one utterance of the utterances comprising a given entity;
evaluating performance of the machine learning model with respect to the given entity using a set of testing data selected from the set of data;
in response to evaluating the performance of the machine learning model, determining that the set of training data should be augmented;
in response to determining that the set of training data should be augmented, generating an augmented set of training data by:
selecting a template from a plurality of templates associated with the given entity;
identifying a plurality of slots in the template, wherein the plurality of slots comprises a first slot that represents a value element for the given entity and a second slot that represents a function element for the given entity;
generating, based on an augmentation rate, a plurality of artificial utterances, wherein the plurality of artificial utterances is generated based on: (i) inserting a value selected from a list of values for the given entity into the first slot; and (ii) inserting a function output obtained by executing a function selected from a plurality of functions associated with the given entity into the second slot, wherein each value of the list of values representing a variation of the given entity, and wherein a number of the plurality of artificial utterances to be generated is determined based on the augmentation rate; and
adding the plurality of artificial utterances to the set of training data; and
using the augmented set of training data to further train the machine learning model to result in the trained machine learning model;
using the entity information identify a chatbot of the one or more chatbots; and
routing the utterance to the chatbot.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
converting training utterances selected from the set of training data that comprise the given entity and context for the given entity into the plurality of templates.

### Claim 3 (depends on 2)

The method of claim 2, wherein converting the training utterances into the plurality of templates comprises maintaining the context for the given entity and replacing the given entity in each training utterance of the training utterances with a slot associated with one or more elements for the given entity, and wherein an element of the one or more elements maps to the list of values for the given entity.

### Claim 4 (depends on 1)

The method of claim 1, wherein the template is selected based on a mapping between the given entity and the plurality of templates, and wherein the plurality of artificial utterances is generated by performing the following iteratively:
selecting a selected value from the list of values for the given entity;
inserting the selected value into the first slot;
executing a function selected from the plurality of functions to obtain a selected function output; and
inserting the selected function output into the second slot.

### Claim 5 (depends on 4)

The method of claim 4, wherein the template is selected based on heuristics.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
evaluating performance of the machine learning model trained using the augmented set of training data using an augmented set of testing data selected from the augmented set of training data; and
in response to evaluating the performance of the machine learning model trained using the augmented set of training data, determining that the augmented set of training data should be augmented to include at least one additional variation of the given entity.

### Claim 7 (depends on 1)

The method of claim 1, wherein the augmentation rate is selected based on the performance of the machine learning model, a random selection method, a predefined schema, a distribution of entities in the set of training data selected from the set of data comprising utterances, or any combination thereof.

### Claim 8

A system comprising:
one or more processors; and
a memory coupled to the one or more processors, the memory storing a plurality of instructions executable by the one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:
accessing an utterance received by a chatbot system comprising one or more chatbots;
using a trained machine learning model to extract entity information from the utterance, wherein the trained machine learning model is trained by:
training a machine learning model using a set of training data selected from a set of data comprising utterances, at least one utterance of the utterances comprising a given entity;
evaluating performance of the machine learning model with respect to the given entity using a set of testing data selected from the set of data;
in response to evaluating the performance of the machine learning model, determining that the set of training data should be augmented;
in response to determining that the set of training data should be augmented, generating an augmented set of training data by:
selecting a template from a plurality of templates associated with the given entity;
identifying a plurality of slots in the template, wherein the plurality of slots comprises a first slot that represents a value element for the given entity and a second slot that represents a function element for the given entity;
generating, based on an augmentation rate, a plurality of artificial utterances, wherein the plurality of artificial utterances is generated based on: (i) inserting a value selected from a list of values for the given entity into the first slot; and (ii) inserting a function output obtained by executing a function selected from a plurality of functions associated with the given entity into the second slot, wherein each value of the list of values representing a variation of the given entity, and wherein a number of the plurality of artificial utterances to be generated is determined based on the augmentation rate; and
adding the plurality of artificial utterances to the set of training data; and
using the augmented set of training data to further train the machine learning model to result in the trained machine learning model;
using the entity information identify a chatbot of the one or more chatbots; and
routing the utterance to the chatbot.

### Claim 9 (depends on 8)

The system of claim 8, further comprising:
converting training utterances selected from the set of training data that comprise the given entity and context for the given entity into the plurality of templates.

### Claim 10 (depends on 9)

The system of claim 9, wherein converting the training utterances into the plurality of templates comprises maintaining the context for the given entity and replacing the given entity in each training utterance of the training utterances with a slot associated with one or more elements for the given entity, and wherein an element of the one or more elements maps to the list of values for the given entity.

### Claim 11 (depends on 8)

The system of claim 8, wherein the template is selected based on a mapping between the given entity and the plurality of templates, and wherein the plurality of artificial utterances is generated by performing the following iteratively:
selecting a selected value from the list of values for the given entity;
inserting the selected value into the first slot;
executing a function selected from the plurality of functions to obtain a selected function output; and
inserting the selected function output into the second slot.

### Claim 12 (depends on 11)

The system of claim 11, wherein the template is selected based on heuristics.

### Claim 13 (depends on 8)

The system of claim 8, further comprising:
evaluating performance of the machine learning model trained using the augmented set of training data using an augmented set of testing data selected from the augmented set of training data; and
in response to evaluating the performance of the machine learning model trained using the augmented set of training data, determining that the augmented set of training data should be augmented to include at least one additional variation of the given entity.

### Claim 14 (depends on 8)

The system of claim 8, wherein the augmentation rate is selected based on the performance of the machine learning model, a random selection method, a predefined schema, a distribution of entities in the set of training data selected from the set of data comprising utterances, or any combination thereof.

### Claim 15

A non-transitory computer-readable memory storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to perform processing comprising:
accessing an utterance received by a chatbot system comprising one or more chatbots;
using a trained machine learning model to extract entity information from the utterance, wherein the trained machine learning model is trained by:
training a machine learning model using a set of training data selected from a set of data comprising utterances, at least one utterance of the utterances comprising a given entity;
evaluating performance of the machine learning model with respect to the given entity using a set of testing data selected from the set of data;
in response to evaluating the performance of the machine learning model, determining that the set of training data should be augmented;
in response to determining that the set of training data should be augmented, generating an augmented set of training data by:
selecting a template from a plurality of templates associated with the given entity;
identifying a plurality of slots in the template, wherein the plurality of slots comprises a first slot that represents a value element for the given entity and a second slot that represents a function element for the given entity;
generating, based on an augmentation rate, a plurality of artificial utterances, wherein the plurality of artificial utterances is generated based on: (i) inserting a value selected from a list of values for the given entity into the first slot; and (ii) inserting a function output obtained by executing a function selected from a plurality of functions associated with the given entity into the second slot, wherein each value of the list of values representing a variation of the given entity, and wherein a number of the plurality of artificial utterances to be generated is determined based on the augmentation rate; and
adding the plurality of artificial utterances to the set of training data; and
using the augmented set of training data to further train the machine learning model to result in the trained machine learning model;
using the entity information identify a chatbot of the one or more chatbots; and
routing the utterance to the chatbot.

### Claim 16 (depends on 15)

The non-transitory computer-readable memory of claim 15, further comprising:
converting training utterances selected from the set of training data that comprise the given entity and context for the given entity into the plurality of templates.

### Claim 17 (depends on 16)

The non-transitory computer-readable memory of claim 16, wherein converting the training utterances into the plurality of templates comprises maintaining the context for the given entity and replacing the given entity in each training utterance of the training utterances with a slot associated with one or more elements for the given entity, and wherein an element of the one or more elements maps to the list of values for the given entity.

### Claim 18 (depends on 15)

The non-transitory computer-readable memory of claim 15, wherein the template is selected based on a mapping between the given entity and the plurality of templates, and wherein the plurality of artificial utterances is generated by performing the following iteratively:
selecting a selected value from the list of values for the given entity;
inserting the selected value into the first slot;
executing a function selected from the plurality of functions to obtain a selected function output; and
inserting the selected function output into the second slot.

### Claim 19 (depends on 15)

The non-transitory computer-readable memory of claim 15, further comprising:
evaluating performance of the machine learning model trained using the augmented set of training data using an augmented set of testing data selected from the augmented set of training data; and
in response to evaluating the performance of the machine learning model trained using the augmented set of training data, determining that the augmented set of training data should be augmented to include at least one additional variation of the given entity.

### Claim 20 (depends on 15)

The non-transitory computer-readable memory of claim 15, wherein the augmentation rate is selected based on the performance of the machine learning model, a random selection method, a predefined schema, a distribution of entities in the set of training data selected from the set of data comprising utterances, or any combination thereof.