US 2025/0094712 A1 · MULTI-GRANULAR CLUSTERING-BASED SOLUTION FOR KEY-VALUE CACHE COMPRESSION

Abstract

Key-value (KV) caching accelerates inference in large language models (LLMs) by allowing the attention operation to scale linearly rather than quadratically with the total sequence length. Due to large context lengths in modern LLMs, KV cache size can exceed the model size, which can negatively impact throughput. To address this issue, a multi-granular clustering-based solution for KV cache compression can be implemented. Key tensors and value tensors corresponding unimportant tokens can be approximated using clusters created at different clustering-levels with varying accuracy. Accuracy loss can be mitigated by using proxies produced at finer granularity clustering-level for a subset of attention heads that are more significant. More significant attention heads can have a higher impact on model accuracy than less significant attention heads. Latency is improved by retrieving proxies from a faster memory for a subset of attention heads that are less significant, when impact on accuracy is lower.

Claims20

Dependent claims

An apparatus comprising:

at least one computer processor;
and
one or more memories storing data at different clustering-levels and instructions;
wherein the at least one computer processor, when executing the instructions, is to:
determine that a token is to be pruned based on an importance of the token, the token being a part of a request to a neural network;
calculate a significance score of an attention head of the neural network;
select a clustering-level for the attention head from different clustering-levels based on the significance score;
store a proxy key tensor and a proxy value tensor produced at the clustering-level in a memory of the one or more memories, the proxy key tensor and the proxy value tensor representing a key tensor and a value tensor calculated by the attention head for the token;
and
provide the proxy key tensor and the proxy value tensor to computing logic that is executing one or more operations of the attention head.

7 more elements

Dependencies are inferred from the claim text.

Multi-granular clustering-based solution for key-value cache compression

Life of this patent

Multi-granular clustering-based solution for key-value cache compression

Life of this patent

Abstract

Claims20