US 12,608,563 B2 · Guided dynamic construction of large language model (LLM) prompts

Abstract

Example solutions for reducing the likelihood of hallucinations by large language models (LLMs) and improving the relevance of generated text are disclosed. A set of multiple of machine learning (ML) models, each custom-trained to identify a different specific topic within text, dynamically generate multiple topic-specific prompts from a received email in an email thread. A user, responding to the prompts, produces material that is used in conjunction with customer relations management (CRM) data and enterprise suite data (e.g., calendar/schedule information) to dynamically generate an LLM prompt. An LLM, using the prompt, generates output text suitable for a business correspondence, such as a reply email, to the sender of the email. The combination of the responses to the multiple topic-specific prompts and the CRM and enterprise suite data in the LLM prompt both ensures relevance of the reply correspondence and minimizes the risk of hallucinations.

Claims20

Dependent claims

A system comprising:

a processor;
and
a computer-readable medium storing instructions that are operative upon execution by the processor to:
receive a first electronic document associated with a first source;
dynamically generate, from the first electronic document, a plurality of topic-specific prompts using a plurality of machine learning (ML) models, each ML model of the plurality of ML models being differently-trained to identify a different specific topic within text;
receive, for each topic-specific prompt of the plurality of topic-specific prompts, a response passage;
dynamically generate a language model prompt using customer relations management (CRM) data associated with the first source, enterprise suite data associated with the first source, and the response passage for the plurality of topic-specific prompts;
dynamically generate an output text using a language model and the language model prompt;
display the plurality of topic-specific prompts in a user interface (UI);
display the output text in the UI;
and
annotate the output text in the UI to indicate elements in the output text having traceability to an associated input source.

9 more elements

A computer storage device having computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising:

receiving a first electronic document, comprising a first email, associated with a first source across a computer network;
dynamically generating, from the first electronic document, a plurality of topic-specific prompts using a plurality of machine learning (ML) models, each ML model of the plurality of ML models being differently-trained to identify a different specific topic within text;
receiving, for each topic-specific prompt of the plurality of topic-specific prompts, a response passage;
dynamically generating a language model prompt using customer relations management (CRM) data associated with the first source, enterprise suite data associated with the first source, and the response passages for the plurality of topic-specific prompts;
dynamically generating an output text using a language model and the language model prompt;
displaying the plurality of topic-specific prompts in a user interface (UI);
displaying the output text in the UI;
and
annotating the output text in the UI to indicate elements in the output text having traceability to an associated input source.

6 more elements

Dependencies are inferred from the claim text.

Guided dynamic construction of large language model (LLM) prompts

Life of this patent

Guided dynamic construction of large language model (LLM) prompts

Life of this patent

Abstract

Claims20