US 12,754,903 B1Granted

Reducing the input prompt size for a large language model

AssigneeRed Hat, Inc.

InventorsMario Fusco, Luca Molteni

Life of this patent

Estimating the term…

Loading patent…

Abstract

The size of an input prompt for a large language model (LLM) can be reduced according to some techniques described herein. As one example, a system can generate a first vector embedding of a natural language query from a user. The system can also generate a second vector embedding of a conversation history between the user and a LLM. The system can then generate a truncated embedding by executing a filtering process based on the first vector embedding and the second vector embedding. The system can generate an input prompt based on the truncated embedding. The system can then execute the LLM based on the input prompt to generate a response to the query. Reducing the size of the input prompt can decrease latency, execution time, and hallucinations associated with the LLM.

Claims20

Dependent claims

A non-transitory computer-readable medium comprising program code that is executable by one or more processors for causing the one or more processors to perform operations including:

receiving a natural language query from a user during a conversation between the user and a large language model (LLM);
and
based on receiving the natural language query, generating a response to the natural language query at least in part by:
generating a first vector embedding of the natural language query from the user;
generating a second vector embedding of a conversation history associated with the conversation;
after generating the first vector embedding and the second vector embedding, generating a truncated embedding based on the second vector embedding, wherein generating the truncated embedding involves executing a filtering process that excludes a portion of the second vector embedding from the truncated embedding based on a comparison between the portion of the second vector embedding and at least a portion of the first vector embedding;
generating an input prompt based on the truncated embedding;
and
executing the LLM based on the input prompt to generate the response to the natural language query.

6 more elements

A method comprising:

receiving, by one or more processors, a natural language query from a user during a conversation between the user and a large language model (LLM);
and
based on receiving the natural language query, generating, by the one or more processors, a response to the natural language query at least in part by:
generating, by one or more processors, a first vector embedding of the natural language query from the user;
generating, by the one or more processors, a second vector embedding of a conversation history associated with the conversation;
after generating the first vector embedding and the second vector embedding, generating, by the one or more processors, a truncated embedding based on the second vector embedding, wherein generating the truncated embedding involves executing a filtering process that excludes a portion of the second vector embedding from the truncated embedding based on a comparison between the portion of the second vector embedding and at least a portion of the first vector embedding;
generating, by the one or more processors, an input prompt based on the truncated embedding;
and
executing, by the one or more processors, the LLM based on the input prompt to generate the response to the natural language query.

6 more elements

A system comprising:

one or more processors;
and
one or more memories storing instructions that are executable by the one or more processors for causing the one or more processors to perform operations including:
receiving a natural language query from a user during a conversation between the user and a large language model (LLM);
and
based on receiving the natural language query, generating a response to the natural language query at least in part by:
generating a first vector embedding of the natural language query from the user;
generating a second vector embedding of a conversation history associated with the conversation;
after generating the first vector embedding and the second vector embedding, generating a truncated embedding based on the second vector embedding, wherein generating the truncated embedding involves executing a filtering process that excludes a portion of the second vector embedding from the truncated embedding based on a comparison between the portion of the second vector embedding and at least a portion of the first vector embedding;
generating an input prompt based on the truncated embedding;
and
executing the LLM based on the input prompt to generate the response to the natural language query.

9 more elements

Dependencies are inferred from the claim text.