US 12,614,080 B2 · Using sample question embeddings to choose between an LLM interfacing model and a non-LLM interfacing model

Abstract

Methods, systems, and computer program products for managing interactions between a content management system (CMS) and a large language model (LLM) system. The semantics of user questions can be considered before prompting an LLM, or alternatively, before querying datasets that are local to the CMS. Given a user question to be answered, the embedding of the user question can be matched against preconfigured sample question embeddings to determine a best match. A prompt corresponding to the determined best match is then configured based on identification of the class or classes that correspond to the matched question. Prompts for provision to LLMs can be synthesized based on a particular user's identity and/or based on the particular user's historical collaboration activities over objects of the CMS. The LLM can be hosted by a third-party provider. Alternatively all or portions of a large language model system can be hosted within the CMS.

Claims20

Dependent claims

A method for choosing between a large language model agent (LLM agent) or a non-LLM agent, the method comprising:

configuring a content management system (CMS) to expose content objects stored in a shared content repository to a plurality of content management system users;
receiving a user question from a content management system user of the plurality of content management system users;
responsive to receipt of the user question, selecting at least one computing agent from a plurality of candidate computing agents, wherein the plurality of candidate computing agents comprise at least a LLM agent and a non-LLM agent, and the at least one computing agent is selected at least based on results of processing the user question to compare the user question to data stored in a database that represents example questions and is separate from the shared content repository;
and
processing the user question by:
when the LLM agent is selected, forming an answer to the user question at least by formulating an LLM prompt using the LLM agent, and sending the LLM prompt to an LLM system, and
when the non-LLM agent is selected, deriving an answer to the user question based on at least characteristics of CMS data associated with at least a subset of the content objects stored in the shared content repository.

4 more elements

A non-transitory computer readable medium having stored thereon a sequence of instructions which, when stored in memory and executed by one or more processors causes the one or more processors to perform a set of acts for choosing between a large language model agent (LLM agent) or a non-LLM agent, the set of acts comprising:

configuring a content management system (CMS) to expose content objects stored in a shared content repository to a plurality of content management system users;
receiving a user question from a content management system user of the plurality of content management system users;
responsive to receipt of the user question, selecting at least one computing agent from a plurality of candidate computing agents, wherein the plurality of candidate computing agents comprise at least a LLM agent and a non-LLM agent, and the at least one computing agent is selected at least based on results of processing the user question to compare the user question to data stored in a database that represents example questions and is separate from the shared content repository;
and
processing the user question by:
when the LLM agent is selected, forming an answer to the user question at least by formulating an LLM prompt using the LLM agent, and sending the LLM prompt to an LLM system, and
when the non-LLM agent is selected, deriving an answer to the user question based on at least characteristics of CMS data associated with at least a subset of the content objects stored in the shared content repository.

4 more elements

Dependencies are inferred from the claim text.

Using sample question embeddings to choose between an LLM interfacing model and a non-LLM interfacing model

Life of this patent

Using sample question embeddings to choose between an LLM interfacing model and a non-LLM interfacing model

Life of this patent

Abstract

Claims20