US 12,626,695 B1Granted

Cache techniques for large language model processing

AssigneeAmazon Technologies, Inc.

InventorsKartik Balasubramaniam, Venkata Siva Sai Krishna Balakavi, Austin Doolittle

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques for cache management for reducing latency in LLM inferencing are described. In some embodiments, a system caches encoded data of portions of a prompt so that the encoded data is available for use by the LLM across dialog turns of a dialog session. Within a dialog session, a portion of the LLM prompt may be the same across dialog turns, and instead of recomputing the attention/encodings for such portions, the cached encodings can be used by the LLM during processing. In some embodiments, user inputs for the dialog session may be routed to the same LLM container and encoded data for the dialog session may be stored at the same cache associated with the LLM. In some embodiments, the system enables asynchronous prompt encoding while performing ASR processing.

Claims20

Dependent claims

A computer-implemented method comprising:

receiving first audio data representing a first user input of a dialog session associated with a session identifier;
determining, using the first audio data, a first transcript representing the first user input, the first transcript associated with the session identifier;
determining first context data corresponding to the first user input;
generating a first prompt including at least the first transcript and the first context data, the first prompt being a first input for a large language model (LLM) to determine a response to the first user input;
selecting, from a group of LLMs, a first LLM to process the first prompt, the first LLM associated with a container identifier for a container that includes the first LLM and corresponding components to enable processing by the first LLM;
determining, using the first LLM, first encoded representations corresponding to the first prompt;
storing, at a cache associated with the first LLM, the first encoded representations;
based on storing the first encoded representations at the cache, storing first data representing an association between the session identifier and the container identifier;
determining, using the first LLM and the first encoded representations, a first response to the first user input;
causing presentation of the first response;
receiving second audio data representing a second user input of the dialog session, the second audio data associated with the session identifier;
determining, using the second audio data, a second transcript representing the second user input, the second transcript associated with the session identifier;
generating a second prompt including at least the first context data, the first transcript, the first response and the second transcript, the second prompt being a second input for a LLM to determine a response to the second user input;
based on the first data and based on the second user input being associated with the session identifier, selecting, from the group of LLMs, the first LLM to process the second prompt;
determining, from the cache associated with the first LLM, the first encoded representations corresponding to a first portion of the second prompt;
determining, using the first LLM, second encoded representations corresponding to a second portion of the second prompt;
storing, at the cache, the second encoded representations;
determining, using the first LLM, the first encoded representations and the second encoded representations, a second response to the second user input;
and
causing presentation of the second response.

17 more elements

A computer-implemented method comprising:

receiving first input data corresponding to a first user input associated with a first session identifier;
generating a first prompt including at least the first input data;
determining, using a first LLM from a group of LLMs, first encoded data corresponding to the first prompt, the first LLM associated with a container identifier for a container that includes the first LLM and corresponding components to enable processing by the first LLM;
storing, at a first cache associated with the first LLM, the first encoded data;
based on storing the first encoded data at the first cache, storing first data representing an association between the first session identifier and the container identifier;
receiving second input data corresponding to a second user input associated with the first session identifier;
generating a second prompt including at least the first input data and the second input data;
based on the first data and based on the second input data being associated with the first session identifier, selecting, from the group of LLMs, the first LLM to process the second prompt;
determining, using the first LLM and the first cache, second encoded data corresponding to the second prompt;
storing, at the first cache, at least a portion of the second encoded data;
and
using the first LLM and at least the second encoded data, determining first output data responsive to the second user input.

9 more elements

A system comprising:

at least one processor;
and
at least one memory including instructions that, when executed by the at least one processor, cause the system to:
receive first input data corresponding to a first user input of a first dialog session, the first input data associated with a first session identifier for the first dialog session;
generate a first prompt including at least the first input data, the first prompt being a first input for a large language model (LLM) to determine a response to the first user input;
determine, using a first LLM from a group of LLMs, first encoded data corresponding to the first prompt, the first LLM being associated with a first container identifier;
store, at a first cache associated with the first LLM, the first encoded data;
based on storing the first encoded data at the first cache, store first data representing a first association between the first dialog session and the first container identifier;
receive second input data corresponding to a second user input of the first dialog session, the second input data associated with the first session identifier;
generate a second prompt including at least the first input data and the second input data, the second prompt being a second input for a LLM to determine a response to the second user input;
based on the first data indicating an association between the first session identifier and the first container identifier and based on the second input data being associated with the first session identifier, select from the group of LLMs, the first LLM to process the second prompt;
determine, using the first LLM and the first cache, second encoded data corresponding to the second prompt;
store, at the first cache, at least a portion of the second encoded data;
and
use the first LLM and at least the second encoded data, determining first output data responsive to the second user input.

12 more elements

Dependencies are inferred from the claim text.