US 12,222,992 B1Granted

Using intent-based rankings to generate large language model responses

AssigneeCitibank, N.A.

InventorsAlberto Cetoli, Jason Ryan Engelbrecht, Youval Bitner, Joel Branch, John E. Ortega

Life of this patent

Estimating the term…

Loading patent…

Abstract

The systems and methods disclosed herein generates responses generated by artificial intelligence (AI) models such as large language models (LLM) using intent-based rankings of retrieved information. The systems and methods disclosed herein receives an output generation request for the generation of an output using a set of AI models. Using a first AI model, a set of documents are retrieved using the received output generation request. The set of documents are partitioned into chunks. The chunks are ranked using a distance between the vector representation of the received output generation request and the vector representation of each chunk. A second AI model classifies the output generation request and chunks using an intent of the respective output generation request or chunk, and generates a second set of rankings using the intents. The set of AI models generate a response using the second set of rankings.

Claims20

Dependent claims

A computer-implemented method for generating responses generated by large language models (LLM) using intent-based rankings of retrieved information, the computer-implemented method comprising:

receiving, from a computing device, an output generation request comprising a prompt for generation of an output using a set of LLMs;
using a first LLM of the set of LLMs, retrieving a set of documents responsive to the prompt in the received output generation request;
partitioning the retrieved set of documents into a set of chunks, wherein each chunk of the set of chunks satisfies a predefined size threshold;
generating, by the first LLM, a first set of rankings of the set of chunks by:
generating a set of vector representations of (i) the received output generation request and (ii) each chunk of the set of chunks of the retrieved set of documents;
and
for each particular chunk in the set of chunks:
determining a distance metric value between the vector representation of the received output generation request and the vector representation of the particular chunk;
and
using the distance metric value between the vector representation of the received output generation request and the vector representation of the particular chunk, assigning the particular chunk a ranking within the first set of rankings;
using a second LLM of the set of LLMs, classifying (i) the received output generation request and (ii) each particular chunk of the set of chunks with a categorical label indicating attributes of one or more of:
(i) an expected output of the received output generation request or (ii) an expected output generation request of the particular chunk;
generating, by the second LLM, a second set of rankings of the set of chunks using the classified categorical labels of the received output generation request and each chunk of the set of chunks,
wherein, in the second set of rankings, chunks with categorical labels matching the categorical label of the received output generation request are ranked higher than chunks with categorical labels failing to match the categorical label of the received output generation request;
and
using (1) the second set of rankings of the set of chunks and (2) information in the retrieved set of documents, automatically generating, by the set of LLMs, a response responsive to the prompt of the received output generation request, wherein the response is configured to be displayed at a user interface of the computing device.

13 more elements

A non-transitory computer-readable storage medium comprising instructions thereon, wherein the instructions when executed by at least one data processor of a system, cause the system to:

receive, from a computing device, an output generation request for generation of an output using a set of artificial intelligence (AI) models;
using a first AI model of the set of AI models, retrieve a set of documents responsive to the received output generation request;
partition the retrieved set of documents into a set of chunks, wherein each chunk of the set of chunks satisfies a set of predetermined criteria;
generate, by the first AI model, a first set of rankings of the set of chunks by:
generating a set of vector representations of (i) the received output generation request and (ii) each chunk of the set of chunks of the retrieved set of documents;
and
for each particular chunk in the set of chunks, using a distance metric value between the vector representation of the received output generation request and the vector representation of the particular chunk, assigning the particular chunk a ranking within the first set of rankings;
using a second AI model of the set of AI models, classify (i) the received output generation request and (ii) each particular chunk of the set of chunks with a categorical label indicating attributes of one or more of:
(i) an expected output of the received output generation request or (ii) an expected output generation request of the particular chunk;
generate, by the second AI model, a second set of rankings of the set of chunks using the classified categorical labels of the received output generation request and each chunk of the set of chunks;
and
using (1) the second set of rankings of the set of chunks and (2) information in the retrieved set of documents, automatically generating, by the set of AI models, a response responsive to the received output generation request.

9 more elements

A system comprising:

at least one hardware processor;
and
at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
obtain, from a set of computing devices, (i) an output generation request for generation of an output using a set of artificial intelligence (AI) models and (ii) a set of documents responsive to the obtained output generation request;
generate, by a first AI model, a first set of rankings of a set of vector representations of (i) the obtained output generation request and (ii) each document of the obtained set of documents using a distance metric value between the vector representation of the obtained output generation request and the vector representation of a particular document of the obtained set of documents;
using a second AI model of the set of AI models, classify (i) the obtained output generation request and (ii) each particular document of the set of documents with a categorical label indicating attributes of one or more of:
(i) an expected output of the obtained output generation request or (ii) an expected output generation request of the particular document;
generate, by the second AI model, a second set of rankings of the set of documents using the classified categorical labels of the obtained output generation request and each document of the set of documents;
and
using (1) the second set of rankings of the set of documents and (2) information in the obtained set of documents, automatically generating, by the set of AI models, a response responsive to the obtained output generation request.

7 more elements

Dependencies are inferred from the claim text.