US 12,754,902 B2Granted

Adaptive follow-up query generation for applications leveraging language models

AssigneePalo Alto Networks, Inc.

InventorsSathishKumar Alwar, Aleksandr Osipov, Haotian Liu, Anil Kumar Nandamuri

Life of this patent

Estimating the term…

Loading patent…

Abstract

A follow-up query generation system (“the system”) proactively and dynamically generates follow-up queries based on user queries submitted to an application that interfaces with a language model(s). The system searches a knowledge base of various subject matter information to determine if the user query is semantically similar to descriptions of subject matter areas represented in the knowledge base. If the user query matches a description of a subject matter area based on semantic similarity, the system retrieves the corresponding knowledge from the knowledge base and invokes a language model with a prompt to generate the follow up queries based at least partly on the corresponding subject matter information. If knowledge is not found which is semantically similar, the system categorizes the user query and determines one or more related categories. The system queries the knowledge base to retrieve a set of pre-defined follow-up queries within the related category(ies).

Claims20

Dependent claims

A method comprising:

based on obtaining a user query comprising natural language, determining if any of a plurality of subject matter descriptions satisfy a semantic similarity criterion for the user query, wherein the plurality of subject matter descriptions was previously generated based on a plurality of example user queries and correspond to a production database,
wherein each of the plurality of subject matter descriptions comprises a description of subject matter to which the corresponding example user query corresponds,
wherein determining if any of the plurality of subject matter descriptions satisfy the semantic similarity criterion for the user query comprises querying associations between the plurality of subject matter descriptions and corresponding information associated with the plurality of subject matter descriptions;
determining that a first subject matter description of the plurality of subject matter descriptions satisfies the semantic similarity criterion for the user query based on obtaining a result of querying the associations between the plurality of subject matter descriptions and corresponding information associated with the plurality of subject matter descriptions, wherein the result indicates first information associated with the first subject matter description;
generating a set of one or more follow-up queries to the user query comprising natural language based on the first information associated with the first subject matter description and the user query,
wherein generating the set of one or more follow-up queries comprises prompting a language model to generate one or more queries comprising natural language as follow-ups to the user query based on the user query and the information associated with the first subject matter description;
and
indicating the set of follow-up queries in a response to the user query.

5 more elements

One or more non-transitory machine-readable media having program code stored thereon, the program code comprising instructions to:

determine whether any of a plurality of subject matter descriptions satisfy a semantic similarity criterion for an obtained user query comprising natural language, wherein each of the plurality of subject matter descriptions was previously generated based on a corresponding one of a plurality of example user queries corresponding to a production database and comprises a description of subject matter to which the corresponding example user query corresponds,
wherein the instructions to determine whether any of the plurality of subject matter descriptions satisfy the semantic similarity criterion comprise instructions to query stored associations between the plurality of subject matter descriptions and corresponding information associated with the plurality of subject matter descriptions with an indication of the user query;
determine that a first subject matter description of the plurality of subject matter descriptions satisfies the semantic similarity criterion based on a result of querying the stored associations between the plurality of subject matter descriptions and corresponding information associated with the plurality of subject matter descriptions;
generate a first set of follow-up queries to the user query comprising natural language based on information associated with the first subject matter description retrieved from querying the stored associations and the user query, wherein the instructions to generate the first set of follow-up queries comprise instructions to prompt a language model to generate one or more queries as follow-ups to the user query based on the information associated with the first subject matter description;
and
provide the first set of follow-up queries in a response to the user query.

3 more elements

An apparatus comprising:

a processor;
and
a machine-readable medium having instructions stored thereon that are executable by the processor to cause the apparatus to,
determine, based on an obtained user query comprising natural language, if any of a plurality of subject matter descriptions satisfy a semantic similarity criterion for the user query, wherein the instructions executable by the processor to cause the apparatus to determine if any of the plurality of subject matter descriptions satisfy the semantic similarity criterion comprise instructions executable by the processor to cause the apparatus to query a database that maintains the plurality of subject matter descriptions and corresponding information associated with the plurality of subject matter descriptions with an indication of the user query,
wherein the plurality of subject matter descriptions and corresponding information were previously generated based on a plurality of example user queries and correspond to a production database,
wherein each of the plurality of subject matter descriptions comprises a description of subject matter to which the corresponding example user query corresponds;
determine that a first subject matter description of the plurality of subject matter descriptions satisfies the semantic similarity criterion for the user query based on obtaining a result of querying the database that comprises information associated with the first subject matter description;
generate a first set of follow-up queries to the user query comprising natural language based on the information associated with the first subject matter description and the user query,
wherein the instructions executable by the processor to cause the apparatus to generate the first set of follow-up queries comprise instructions executable by the processor to cause the apparatus to prompt a language model to generate one or more follow-up queries to the user query based on the information associated with the first subject matter description;
and
indicate the first set of follow-up queries in a response to the user query.

8 more elements

Dependencies are inferred from the claim text.