US 2026/0300283 A1Published application

Data augmentation pipeline for multi-turn text-to-sql

AssigneeOracle International Corporation

InventorsBhagya Gayathri Hettige, Qianyu Feng, Varsha Kuppur Rajendra, Mohammad Najafi, Dalu Guo, Duy Vu, Yakupitiyage Don Thanuja Samodhye Dharmasiri

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

Techniques are provided for generating context-aware, multi-turn natural language to SQL training data for database query models. The method involves accessing reference data that includes natural language utterances and corresponding queries in a programming query language. An AI model may generate a set of follow-up natural language utterances based on contextual information related to an initial utterance, and may rewrite these utterances to produce self-contained, multi-turn queries. An AI model can retrieve examples from the reference data that are semantically similar to the multi-turn utterances, serving as in-context examples. An AI model may then generate candidate queries based on the rewritten utterances and the retrieved examples. A dual stage validation process can be executed to evaluate the candidate queries, generating performance reports indicating which queries pass validation. Multi-turn training data comprising the validated utterances and their corresponding queries may then be generated for use in training database query models.

Claims20

Dependent claims

A computer-implemented method comprising:

accessing reference data comprising natural language utterances and corresponding queries in a programming query language;
executing, by one or more generative models, a natural language utterance generation process to generate multi-turn natural language utterances, wherein executing comprises:
generating, by one of the one or more generative models, a set of follow-up natural language utterances based on contextual information relating to a natural language utterance in the reference data, and
rewriting, by one of the one or more generative models, the set of follow-up natural language utterances to generate self-contained multi-turn natural language utterances;
executing, by one or more artificial intelligence (AI) models, a query generation process to generate candidate queries corresponding to the multi-turn natural language utterances, wherein executing comprises:
retrieving, by one of the one or more AI models, one or more examples from the reference data having a natural language utterance that is semantically similar to the multi-turn natural language utterances, wherein the retrieved one or more examples are in-context examples,
generating, by one of the one or more AI models, candidate queries based on the multi-turn natural language utterances and the in-context examples, and
executing, using one of the one or more AI models, a dual stage validation process on the candidate queries, wherein the dual stage validation process generates performance reports indicating whether a candidate query passes the dual stage validation process;
and
generating multi-turn natural language to query training data comprising the multi-turn natural language utterances and corresponding candidate queries that pass the dual stage validation process.

7 more elements

A system comprising:

one or more processors;
and
one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations comprising:
(i) accessing reference data comprising natural language utterances and corresponding queries in a programming query language;
(ii) generating, by a generative model, a set of follow-up natural language utterances based on contextual information relating to a natural language utterance in the reference data;
(iii) rewriting, by a generative model, the set of follow-up natural language utterances to generate self-contained multi-turn natural language utterances;
(iv) retrieving, by an embedding model, one or more examples from the reference data having a natural language utterance that is semantically similar to the multi-turn natural language utterances, wherein the retrieved one or more examples are in-context examples;
(v) generating, by a natural language to query model, candidate queries based on the multi-turn natural language utterances and the in-context examples;
(vi) executing, by one or more AI models, a dual stage validation process on the candidate queries, wherein the dual stage validation process generates performance reports indicating whether a candidate query passes the dual stage validation process;
and
(vii) generating multi-turn natural language to query training data comprising the multi-turn natural language utterances and corresponding candidate queries that pass the dual stage validation process.

8 more elements

One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

(i) accessing reference data comprising natural language utterances and corresponding queries in a programming query language;
(ii) generating, by a generative model, a set of follow-up natural language utterances based on contextual information relating to a natural language utterance in the reference data;
(iii) rewriting, by a generative model, the set of follow-up natural language utterances to generate self-contained multi-turn natural language utterances;
(iv) retrieving, by an embedding model, one or more examples from the reference data having a natural language utterance that is semantically similar to the multi-turn natural language utterances, wherein the retrieved one or more examples are in-context examples;
(v) generating, by a natural language to query model, candidate queries based on the multi-turn natural language utterances and the in-context examples;
(vi) executing, by one or more AI models, a dual stage validation process on the candidate queries, wherein the dual stage validation process generates performance reports indicating whether a candidate query passes the dual stage validation process;
and
(vii) when candidate queries fail a first stage of the dual stage validation process, a second stage of the dual stage validation process, or both:
executing, by a refinement generative model, an iterative refinement loop, wherein at least the performance reports for the failed candidate queries are incorporated into a prompt of the refinement generative model and the iterative refinement loop repeats step (vi) for one or a predetermined number of iterations, or until a candidate query passes the first stage and the second stage of the dual stage validation process;
and
(viii) generating multi-turn natural language to query training data comprising the multi-turn natural language utterances and corresponding candidate queries that pass the dual stage validation process.

8 more elements

Dependencies are inferred from the claim text.