Key value pair cache management

A system, comprising:

• a memory unit, capable of storing one or more key-value (KV) pairs in at least one KV cache;
• a processing unit capable of executing code to decode a source material using a large language model (LLM) and the one or more KV pairs, wherein the processing unit is communicatively coupled to the memory unit;
• and
• a KV management unit capable of managing the one or more KV pairs stored in the at least one KV cache, wherein the KV management unit utilizes a KV cache eviction algorithm and a dynamic KV token selection algorithm, wherein the dynamic KV token selection algorithm uses a hybrid attention process and the KV cache eviction algorithm uses an adaptive pooling size.

A non-transitory computer program product having a series of operating instructions stored on a non-transitory computer-readable medium that directs a key-value (KV) management process when executed thereby to perform operations, the operations comprising:

• receiving a source material and input parameters;
• and
• decoding the source material using a large language model (LLM), wherein the decoding includes:
• ascertaining an input sequence of the source material;
• and
• identifying key-value (KV) pairs stored in at least one KV cache, wherein the at least one KV cache is managed using a KV management process comprising of a KV cache eviction algorithm and a dynamic KV token selection algorithm, wherein the KV cache eviction algorithm uses an adaptive pooling size and the dynamic KV token selection algorithm applies a hybrid attention process.

A processing unit, comprising:

• a code execution system, capable of decoding a source material using a large language model (LLM), wherein the LLM utilizes a multi-turn interaction framework;
• a key-value (KV) cache capable of storing one or more KV pairs computed by the LLM for the source material;
• and
• a KV management unit, capable of managing the one or more KV pairs stored in the KV cache, wherein the KV management unit utilizes a KV cache eviction algorithm and a dynamic KV token selection algorithm, wherein the dynamic KV token selection algorithm uses a hybrid attention process and the KV cache eviction algorithm uses an adaptive pooling size.

A processing unit system, comprising:

• a code execution system, capable of decoding a source material using a large language model (LLM);
• a key-value (KV) cache system capable of storing one or more KV pairs used by the LLM;
• and
• a KV management system, capable of managing the one or more KV pairs stored in the KV cache utilizing a KV cache eviction algorithm and a dynamic KV token selection algorithm, wherein the dynamic KV token selection algorithm uses a hybrid attention process and the KV cache eviction algorithm uses an adaptive pooling size.