# US 12,602,617 B2 · Data manufacturing frameworks for synthesizing synthetic training data to facilitate training a natural language to logical form model > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,617 B2 - Document ID: US12602617B2 - Status: Granted - Kind: B2 - Application number: 18/065,406 - Filed: 2022-12-13 - Published: 2023-06-15 - Granted: 2026-04-14 - Earliest claimed: 2021-12-14 - Assignees: Oracle International Corporation - Applicants: Oracle International Corporation - Inventors: Philip Arthur; Vishal Vishnoi; Mark Edward Johnson; Thanh Long Duong; Srinivasa Phani Kumar Gadde; Balakota Srinivas Vinnakota; Cong Duy Vu Hoang; Steve Wai-Chun Siu; Nitika Mathur; Gioacchino Tangari; Aashna Devang Kanuga - Web page: https://patentgrep.com/patent/US12602617B2 ## CPC - G06N 20/00: Machine learning - G06F 16/24522: {Translation of natural language queries to structured queries} - G06F 16/3329: Natural language query formulation - G06F 16/90332: {Natural language query formulation or dialogue systems} - G06F 40/211: Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars - G06F 40/237: Lexical tools - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06F 40/47: Machine-assisted translation, e.g. using translation memory - G06F 40/56: Natural language generation - G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 5/046: Forward inferencing; Production systems - H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages - G06F 40/205: Parsing - G06F 40/35: Discourse or dialogue representation - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0455: Auto-encoder networks; Encoder-decoder networks - G06N 3/096: Transfer learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Techniques are disclosed herein for synthesizing synthetic training data to facilitate training a natural language to logical form model. In one aspect, training data can be synthesized from original under a framework based on templates and a synchronous context-free grammar. In one aspect, training data can be synthesized under a framework based on a probabilistic context-free grammar and a translator. In one aspect, training data can be synthesized under a framework based on tree-to-string translation. In one aspect, the synthetic training data can be combined with original training data in order to train a machine learning model to translate an utterance to a logical form. ## Claims ### Claim 1 A computer-implemented method comprising: accessing original training data, the original training data including a plurality of utterances, a plurality of logical forms, and database schema information, each logical form of the plurality of logical forms corresponding to at least one utterance of the plurality of utterances; generating a plurality of templates, each template of the plurality of templates including a delexicalized version of an utterance in the plurality of utterances and a delexicalized version of a logical form corresponding to the utterance, wherein the delexicalized version of the utterance comprises a non-terminal symbol that represents an element associated with the database schema information; learning a grammar from the plurality of logical forms, the grammar defining a plurality of production rules for lexicalizing the plurality of templates; generating synthetic training data, wherein generating the synthetic training data comprises: for each respective template of the plurality of templates: identifying one or more constraints for respective template based on the non-terminal symbol of the delexicalized version of the utterance for the respective template, sampling a database to identify a sampling component of the database based on the one or more constraints, lexicalizing the delexicalized version of the utterance for the respective template by replacing the non-terminal symbol of the delexicalized version of the utterance for the respective template with the sampling component; and training a machine learning model with the original training data and the synthetic training data to result in a trained machine learning model that is configured to translate utterances to logical forms. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the plurality of templates is generated automatically from the plurality of utterances and the plurality of logical forms using a machine-learning model. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the grammar is a synchronous context-free grammar. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein learning the grammar comprises setting one or more table names, column names, and values in database schema information included in the original training data as non-terminal symbols and generating the plurality of production rules by replacing one or more words, entities, or phrases in the plurality of utterances with the non-terminal symbols. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein generating the synthetic training data further comprises: parsing the plurality of templates by applying a synchronous context-free grammar to delexicalized versions of the utterances of the plurality of templates and the delexicalized versions of the logical forms of the plurality of templates to generate abstract syntax trees for the plurality of templates. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, further comprising: using the trained machine learning model to translate an input utterance into a logical form for the input utterance; and executing the logical form for the input utterance as a query on a database to retrieve a result for the query. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein using the trained machine learning model to translate the input utterance into the logical form for the input utterance comprises: accessing an utterance; inputting the utterance into the trained machine learning model; and translating, using the trained machine learning model, the utterance into the logical form. ### Claim 8 A system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising: accessing original training data, the original training data including a plurality of utterances, a plurality of logical forms, and database schema information, each logical form of the plurality of logical forms corresponding to at least one utterance of the plurality of utterances; generating a plurality of templates, each template of the plurality of templates including a delexicalized version of an utterance in the plurality of utterances and a delexicalized version of a logical form corresponding to the utterance, wherein the delexicalized version of the utterance comprises a non-terminal symbol that represents an element associated with the database schema information; learning a grammar from the plurality of logical forms, the grammar defining a plurality of production rules for lexicalizing the plurality of templates; generating synthetic training data, wherein generating the synthetic training data comprises: for each respective template of the plurality of templates: identifying one or more constraints for respective template based on the non-terminal symbol of the delexicalized version of the utterance for the respective template, sampling a database to identify a sampling component of the database based on the one or more constraints, lexicalizing the delexicalized version of the utterance for the respective template by replacing the non-terminal symbol of the delexicalized version of the utterance for the respective template with the sampling component; and training a machine learning model with the original training data and the synthetic training data to result in a trained machine learning model that is configured to translate utterances to logical forms. ### Claim 9 (depends on 8) The system of claim 8, wherein the plurality of templates is generated automatically from the plurality of utterances and the plurality of logical forms using a machine-learning model. ### Claim 10 (depends on 8) The system of claim 8, wherein the grammar is a synchronous context-free grammar. ### Claim 11 (depends on 8) The system of claim 8, wherein learning the grammar comprises setting one or more table names, column names, and values in database schema information included in the original training data as non-terminal symbols and generating the plurality of production rules by replacing one or more words, entities, or phrases in the plurality of utterances with the non-terminal symbols. ### Claim 12 (depends on 8) The system of claim 8, wherein generating the synthetic training data further comprises: parsing the plurality of templates by applying a synchronous context-free grammar to delexicalized versions of the utterances of the plurality of templates and the delexicalized versions of the logical forms of the plurality of templates to generate abstract syntax trees for the plurality of templates. ### Claim 13 (depends on 8) The system of claim 8, the operations further comprising: using the trained machine learning model to translate an input utterance into a logical form for the input utterance; and executing the logical form for the input utterance as a query on a database to retrieve a result for the query. ### Claim 14 (depends on 13) The system of claim 13, wherein using the trained machine learning model to translate the input utterance into the logical form for the input utterance comprises: accessing an utterance; inputting the utterance into the trained machine learning model; and translating, using the trained machine learning model, the utterance into the logical form. ### Claim 15 A computer-program product tangibly embodied in one or more non-transitory machine-readable media, including instructions configured to cause one or more data processors to perform the following operations: accessing original training data, the original training data including a plurality of utterances, a plurality of logical forms, and database schema information, each logical form of the plurality of logical forms corresponding to at least one utterance of the plurality of utterances; generating a plurality of templates, each template of the plurality of templates including a delexicalized version of an utterance in the plurality of utterances and a delexicalized version of a logical form corresponding to the utterance, wherein the delexicalized version of the utterance comprises a non-terminal symbol that represents an element associated with the database schema information; learning a grammar from the plurality of logical forms, the grammar defining a plurality of production rules for lexicalizing the plurality of templates; generating synthetic training data, wherein generating the synthetic training data comprises: for each respective template of the plurality of templates: identifying one or more constraints for respective template based on the non-terminal symbol of the delexicalized version of the utterance for the respective template, sampling a database to identify a sampling component of the database based on the one or more constraints, lexicalizing the delexicalized version of the utterance for the respective template by replacing the non-terminal symbol of the delexicalized version of the utterance for the respective template with the sampling component; and training a machine learning model with the original training data and the synthetic training data to result in a trained machine learning model that is configured to translate utterances to logical forms. ### Claim 16 (depends on 15) The computer-program product of claim 15, wherein the plurality of templates is generated automatically from the plurality of utterances and the plurality of logical forms using a machine-learning model. ### Claim 17 (depends on 15) The computer-program product of claim 15, wherein the grammar is a synchronous context-free grammar. ### Claim 18 (depends on 15) The computer-program product of claim 15, wherein learning the grammar comprises setting one or more table names, column names, and values in database schema information included in the original training data as non-terminal symbols and generating the plurality of production rules by replacing one or more words, entities, or phrases in the plurality of utterances with the non-terminal symbols. ### Claim 19 (depends on 15) The computer-program product of claim 15, wherein generating the synthetic training data further comprises: parsing the plurality of templates by applying a synchronous context-free grammar to delexicalized versions of the utterances of the plurality of templates and the delexicalized versions of the logical forms of the plurality of templates to generate abstract syntax trees for the plurality of templates. ### Claim 20 (depends on 15) The computer-program product of claim 15, the operations further comprising: using the trained machine learning model to translate an input utterance into a logical form for the input utterance; and executing the logical form for the input utterance as a query on a database to retrieve a result for the query.