# US 2025/0156649 A1 · Model robustness on operators and triggering keywords in natural language to a meaning representation language system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0156649 A1
- Document ID: US20250156649A1
- Status: Published application
- Kind: A1
- Application number: 18/505,498
- Filed: 2023-11-09
- Published: 2025-05-15
- Earliest claimed: 2023-11-09
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Gioacchino Tangari; Chang Xu; Nitika Mathur; Philip Arthur; Syed Najam Abbas Zaidi; Aashna Devang Kanuga; Cong Duy Vu Hoang; Poorya Zaremoodi; Thanh Long Duong; Mark Edward Johnson; Vishal Vishnoi
- Web page: https://patentgrep.com/patent/US20250156649A1

## CPC

- G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30)
- G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
- G06F 40/284: Lexical analysis, e.g. tokenisation or collocates
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Techniques are disclosed herein for improving model robustness on operators and triggering keywords in natural language to a meaning representation language system. The techniques include augmenting an original set of training data for a target robustness bucket by leveraging a combination of two training data generation techniques: (1) modification of existing training examples and (2) synthetic template-based example generation. The resulting set of augmented data examples from the two training data generation techniques are appended to the original set of training data to generate an augmented training data set and the augmented training data set is used to train a machine learning model to generate logical forms for utterances.

## Claims

### Claim 1

A computer-implemented method comprising:
accessing training data comprising a plurality of training examples comprising a first training example, wherein the first training example comprises a first natural language utterance and a first logical form for the first natural language utterance, and wherein the first natural language utterance comprises one or more keywords associated with a target robustness bucket;
replacing, in the first natural language utterance, the one or more keywords with one or more replacement terms sampled from a list of replacement terms to generate a second natural language utterance;
generating a second training example comprising the second natural language utterance and the first logical form;
accessing a data manufacturing template that defines structure of a system operation and constraints imposed on operators of the system operation;
generating a second logical form by filling slots for one or more of the operators in the data manufacturing template based on a schema and values for a system, wherein the one or more of the operators are associated with the target robustness bucket;
translating the second logical form into a third natural language utterance based on a grammar data structure that includes a custom grammar and a set of rules for translating logical form statements into corresponding natural language expressions comprising one or more target keywords, wherein the custom grammar comprises target keywords including the replacement terms from the list of replacement terms, which are used to control the translating such that natural language utterances will contain one or more of the target keywords;
generating a third training example comprising the second logical form and the third natural language utterance;
augmenting the training data by adding the second training example and the third training example to the plurality of training examples to generate an augmented training data set; and
training a machine learning model to generate logical forms for utterances using the augmented training data set.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein accessing the training data comprises identifying, from the plurality of training examples, a set of target training examples that match patterns comprising the one or more keywords or the one or more of the operators associated with the target robustness bucket, wherein the set of target training examples comprise the first training example.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the first training example comprises the first natural language utterance, the first logical form for the first natural language utterance, and metadata associated with the first natural language utterance, wherein generating the second training example comprises modifying the metadata associated with the first natural language utterance to generate updated metadata associated with the second natural language utterance, and wherein modifying the metadata comprises adjusting an offset value for schema linking to reflect values in the second natural language utterance.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the data manufacturing template comprises a delexicalized version of the second logical form and filling the slots comprises: parsing the data manufacturing template, sampling the system and schema to identify a plurality of sampling components comprising the values, and lexicalizing the slots of the data manufacturing template with at least one sampling component of the plurality of sampling components.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the data manufacturing template is parsed by applying a synchronous context-free grammar to the delexicalized version of the second logical form to generate an abstract syntax tree for the data manufacturing template, and lexicalizing the data manufacturing template comprises analyzing the data manufacturing template to identify the constraints imposed on the operators and lexicalizing the slots of the data manufacturing template with at least one sampling component of the plurality of sampling components based on the identified constraints.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein translating the second logical form into a third natural language utterance comprises:
converting the second logical form into a logical form structure comprising one or more logical form statements, wherein the converting comprises extracting one or more operations and associated one or more attributes from the second logical form and generating the one or more logical form statements from the one or more operations and associated one or more attributes;
translating each of the one or more logical form statements into a natural language expression based on the grammar data structure;
combining the one or more natural language expressions into the third natural language utterance; and
providing third natural language utterance as an interpretation of the second logical form.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein training the machine learning model using the augmented training data set comprises:
sampling training values from the augmented training data set based on a sampling rate; and
training the machine learning model using the sampled training values.

### Claim 8

A system comprising:
one or more processors; and
one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations comprising:
accessing training data comprising a plurality of training examples comprising a first training example, wherein the first training example comprises a first natural language utterance and a first logical form for the first natural language utterance, and wherein the first natural language utterance comprises one or more keywords associated with a target robustness bucket;
replacing, in the first natural language utterance, the one or more keywords with one or more replacement terms sampled from a list of replacement terms to generate a second natural language utterance;
generating a second training example comprising the second natural language utterance and the first logical form;
accessing a data manufacturing template that defines structure of a system operation and constraints imposed on operators of the system operation;
generating a second logical form by filling slots for one or more of the operators in the data manufacturing template based on a schema and values for a system, wherein the one or more of the operators are associated with the target robustness bucket;
translating the second logical form into a third natural language utterance based on a grammar data structure that includes a custom grammar and a set of rules for translating logical form statements into corresponding natural language expressions comprising one or more target keywords, wherein the custom grammar comprises target keywords including the replacement terms from the list of replacement terms, which are used to control the translating such that natural language utterances will contain one or more of the target keywords;
generating a third training example comprising the second logical form and the third natural language utterance;
augmenting the training data by adding the second training example and the third training example to the plurality of training examples to generate an augmented training data set; and
training a machine learning model to generate logical forms for utterances using the augmented training data set.

### Claim 9 (depends on 8)

The system of claim 8, wherein accessing the training data comprises identifying, from the plurality of training examples, a set of target training examples that match patterns comprising the one or more keywords or the one or more of the operators associated with the target robustness bucket, wherein the set of target training examples comprise the first training example.

### Claim 10 (depends on 8)

The system of claim 8, wherein the first training example comprises the first natural language utterance, the first logical form for the first natural language utterance, and metadata associated with the first natural language utterance, wherein generating the second training example comprises modifying the metadata associated with the first natural language utterance to generate updated metadata associated with the second natural language utterance, and wherein modifying the metadata comprises adjusting an offset value for schema linking to reflect values in the second natural language utterance.

### Claim 11 (depends on 8)

The system of claim 8, wherein the data manufacturing template comprises a delexicalized version of the second logical form and filling the slots comprises: parsing the data manufacturing template, sampling the system and schema to identify a plurality of sampling components comprising the values, and lexicalizing the slots of the data manufacturing template with at least one sampling component of the plurality of sampling components.

### Claim 12 (depends on 11)

The system of claim 11, wherein the data manufacturing template is parsed by applying a synchronous context-free grammar to the delexicalized version of the second logical form to generate an abstract syntax tree for the data manufacturing template, and lexicalizing the data manufacturing template comprises analyzing the data manufacturing template to identify the constraints imposed on the operators and lexicalizing the slots of the data manufacturing template with at least one sampling component of the plurality of sampling components based on the identified constraints.

### Claim 13 (depends on 8)

The system of claim 8, wherein translating the second logical form into a third natural language utterance comprises:
converting the second logical form into a logical form structure comprising one or more logical form statements, wherein the converting comprises extracting one or more operations and associated one or more attributes from the second logical form and generating the one or more logical form statements from the one or more operations and associated one or more attributes;
translating each of the one or more logical form statements into a natural language expression based on the grammar data structure;
combining the one or more natural language expressions into the third natural language utterance; and
providing third natural language utterance as an interpretation of the second logical form.

### Claim 14 (depends on 8)

The system of claim 8, wherein training the machine learning model using the augmented training data set comprises:
sampling training values from the augmented training data set based on a sampling rate; and
training the machine learning model using the sampled training values.

### Claim 15

One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause a system to perform operations comprising:
accessing training data comprising a plurality of training examples comprising a first training example, wherein the first training example comprises a first natural language utterance and a first logical form for the first natural language utterance, and wherein the first natural language utterance comprises one or more keywords associated with a target robustness bucket;
replacing, in the first natural language utterance, the one or more keywords with one or more replacement terms sampled from a list of replacement terms to generate a second natural language utterance;
generating a second training example comprising the second natural language utterance and the first logical form;
accessing a data manufacturing template that defines structure of a system operation and constraints imposed on operators of the system operation;
generating a second logical form by filling slots for one or more of the operators in the data manufacturing template based on a schema and values for a system, wherein the one or more of the operators are associated with the target robustness bucket;
translating the second logical form into a third natural language utterance based on a grammar data structure that includes a custom grammar and a set of rules for translating logical form statements into corresponding natural language expressions comprising one or more target keywords, wherein the custom grammar comprises target keywords including the replacement terms from the list of replacement terms, which are used to control the translating such that natural language utterances will contain one or more of the target keywords;
generating a third training example comprising the second logical form and the third natural language utterance;
augmenting the training data by adding the second training example and the third training example to the plurality of training examples to generate an augmented training data set; and
training a machine learning model to generate logical forms for utterances using the augmented training data set.

### Claim 16 (depends on 15)

The one or more non-transitory computer-readable media of claim 15, wherein accessing the training data comprises identifying, from the plurality of training examples, a set of target training examples that match patterns comprising the one or more keywords or the one or more of the operators associated with the target robustness bucket, wherein the set of target training examples comprise the first training example.

### Claim 17 (depends on 15)

The one or more non-transitory computer-readable media of claim 15, wherein the first training example comprises the first natural language utterance, the first logical form for the first natural language utterance, and metadata associated with the first natural language utterance, wherein generating the second training example comprises modifying the metadata associated with the first natural language utterance to generate updated metadata associated with the second natural language utterance, and wherein modifying the metadata comprises adjusting an offset value for schema linking to reflect values in the second natural language utterance.

### Claim 18 (depends on 15)

The one or more non-transitory computer-readable media of claim 15, wherein the data manufacturing template comprises a delexicalized version of the second logical form and filling the slots comprises: parsing the data manufacturing template, sampling the system and schema to identify a plurality of sampling components comprising the values, and lexicalizing the slots of the data manufacturing template with at least one sampling component of the plurality of sampling components.

### Claim 19 (depends on 18)

The one or more non-transitory computer-readable media of claim 18, wherein the data manufacturing template is parsed by applying a synchronous context-free grammar to the delexicalized version of the second logical form to generate an abstract syntax tree for the data manufacturing template, and lexicalizing the data manufacturing template comprises analyzing the data manufacturing template to identify the constraints imposed on the operators and lexicalizing the slots of the data manufacturing template with at least one sampling component of the plurality of sampling components based on the identified constraints.

### Claim 20 (depends on 15)

The one or more non-transitory computer-readable media of claim 15, wherein translating the second logical form into a third natural language utterance comprises:
converting the second logical form into a logical form structure comprising one or more logical form statements, wherein the converting comprises extracting one or more operations and associated one or more attributes from the second logical form and generating the one or more logical form statements from the one or more operations and associated one or more attributes;
translating each of the one or more logical form statements into a natural language expression based on the grammar data structure;
combining the one or more natural language expressions into the third natural language utterance; and
providing third natural language utterance as an interpretation of the second logical form.
