Automated subset-match evaluation for normalized nl2sql prompts and generative models

A system comprising:

• one or more processors;
• and
• one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform operations comprising:
• obtaining a query, wherein the query is associated with a natural language utterance;
• generating, by a generative model, a logical form query that corresponds to the natural language utterance;
• generating a normalized logical form query by modifying the logical form query based on a schema;
• executing the normalized logical form query on a data store to obtain results;
• identifying whether the results include at least reference results associated with execution of a reference logical form query on the data store;
• and
• indicating, in response to identifying that the results include at least the reference results, that the logical form query is acceptable.

One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• obtaining a query, wherein the query is associated with a natural language utterance;
• generating, by a generative model, a logical form query that corresponds to the natural language utterance;
• generating a normalized logical form query by modifying the logical form query based on a schema;
• executing the normalized logical form query on a data store to obtain results;
• identifying whether the results include at least reference results associated with execution of a reference logical form query on the data store;
• and
• indicating, in response to identifying that the results include at least the reference results, that the logical form query is acceptable.