# US 2026/0094232 A1 · Systems and methods for key-value (kv) cache pruning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0094232 A1
- Document ID: US20260094232A1
- Status: Published application
- Kind: A1
- Application number: 19/040,001
- Filed: 2025-01-29
- Published: 2026-04-02
- Earliest claimed: 2024-10-01
- Assignees: Salesforce, Inc.
- Applicants: Salesforce, Inc.
- Inventors: Yuhui Xu
- Web page: https://patentgrep.com/patent/US20260094232A1

## CPC

- G06T 1/60: Memory management
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Embodiments described herein provide a key-value (KV) cache pruning framework to improve hardware efficiency while maintaining computational accuracy of large language models (LLMS). Specifically, the channel dimension D of a key cache (or value cache) may be pruned by dynamically identifying unimportant channels based on data dependent criterion and abstracting away identified redundancies in each head's key cache (or value cache). The framework is orthogonal to other KV cache compression schemes (e.g., KV cache eviction, quantization) and can complement (without incurring loss) these other schemes. Therefore, with such improved memory optimization in KV caching, neural network technology in LLMs is improved.

## Claims

### Claim 1

A method of managing cache usage on a graphic processing unit (GPU) for a Transformer-based neural network model, the method comprising:
transforming, by the Transformer-based neural network model implemented on one or more processors, an input sequence of tokens into intermediate variables including at least a key matrix, a value matrix, and a query matrix;
allocating, in one or more processor memories, a key cache for storing the key matrix and a value cache for storing the value matrix;
generating a plurality of scores indicating magnitudes associated with a plurality of rows of the key matrix stored at the key cache, respectively;
re-allocating at least a portion of the key cache by removing at least one or more rows of the key matrix having associated scores below a score threshold, each row of the key matrix corresponding to a channel of the key cache; and
operating the Transformer-based neural network model on the one or more processors with a reduced key cache after the re-allocating.

### Claim 2 (depends on 1)

The method of claim 1, wherein the scores associated with each row in the key cache are attention scores computed using queries and keys in the respective query matrix and key matrix, wherein the re-allocating of the key cache filters the key matrix by only keeping rows in the key matrix that have attention scores higher than an attention score threshold.

### Claim 3 (depends on 2)

The method of claim 2, wherein the key matrix is filtered through a channel mask matrix also stored in the key cache.

### Claim 4 (depends on 2)

The method of claim 2, wherein the re-allocating further includes performing low-rank approximation on the key matrix based on the attention scores.

### Claim 5 (depends on 1)

The method of claim 1, wherein the scores associated with each row in the key cache are absolute magnitude values of keys in the key matrix, wherein the re-allocating of the key cache filters the key matrix by only keeping channels in the rows that have magnitude values higher than a magnitude score threshold.

### Claim 6 (depends on 5)

The method of claim 5, wherein the scores are first scores and the score threshold is a first score threshold, further comprising:
generating a plurality of second scores indicating magnitudes associated with a plurality of rows of the value matrix stored at the value cache, respectively;
re-allocating at least a portion of the value cache by removing at least one or more rows of the value matrix having associated second scores below a second score threshold, each row of the value matrix corresponding to a channel of the value cache; and
operating the Transformer-based neural network model on the one or more processors with a reduced value cache.

### Claim 7 (depends on 6)

The method of claim 6, wherein the second scores associated with each row in the value cache are scores computed by multiplying respective values in the value matrix with attention scores calculated using queries and keys in the respective query matrix and key matrix, wherein the re-allocating of the value cache filters the value matrix by only keeping rows in the value matrix that have second scores higher than the second score threshold.

### Claim 8 (depends on 1)

The method of claim 1,
wherein the key cache has a cache size defined by dimensions B×S×L×N×D, where B is a batch size of the input sequence, S is a sequence length of the input sequence, L is a total number of layer in the Transformer-based neural network model, N is a number of heads in each layer, and D is a total number of channels in each head,
wherein the removing of the at least one or more rows of the key matrix prunes one or more channels from the dimension D,
further comprising performing cache eviction and/or structured pruning techniques to prune the key cache from the dimension S or the dimension L.

### Claim 9

A system for managing cache usage on a graphic processing unit (GPU) for a Transformer-based neural network model, the system comprising:
a memory that stores a Transformer-based neural network model and a plurality of processor-executable instructions;
a communication interface that receives an input sequence of tokens; and
one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
transforming, by the Transformer-based neural network model, the input sequence of tokens into intermediate variables including at least a key matrix, a value matrix, and a query matrix;
allocating, in one or more processor memories, a key cache for storing the key matrix and a value cache for storing the value matrix;
generating a plurality of scores indicating magnitudes associated with a plurality of rows of the key matrix stored at the key cache, respectively;
re-allocating at least a portion of the key cache by removing at least one or more rows of the key matrix having associated scores below a score threshold, each row of the key matrix corresponding to a channel of the key cache; and
operating the Transformer-based neural network model on the one or more processors with a reduced key cache after the re-allocating.

### Claim 10 (depends on 9)

The system of claim 9, wherein the scores associated with each row in the key cache are attention scores computed using queries and keys in the respective query matrix and key matrix, wherein the re-allocating of the key cache filters the key matrix by only keeping rows in the key matrix that have attention scores higher than an attention score threshold.

### Claim 11 (depends on 9)

The system of claim 9, wherein the scores associated with each row in the key cache are absolute magnitude values of keys in the key matrix, wherein the re-allocating of the key cache filters the key matrix by only keeping channels in the rows that have magnitude values higher than a magnitude score threshold.

### Claim 12 (depends on 9)

The system of claim 9, wherein the scores are first scores and the score threshold is a first score threshold, further comprising:
generating a plurality of second scores indicating magnitudes associated with a plurality of rows of the value matrix stored at the value cache, respectively;
re-allocating at least a portion of the value cache by removing at least one or more rows of the value matrix having associated second scores below a second score threshold, each row of the value matrix corresponding to a channel of the value cache; and
operating the Transformer-based neural network model on the one or more processors with a reduced value cache.

### Claim 13 (depends on 12)

The system of claim 12, wherein the second scores associated with each row in the value cache are scores computed by multiplying respective values in the value matrix with attention scores calculated using queries and keys in the respective query matrix and key matrix, wherein the re-allocating of the value cache filters the value matrix by only keeping rows in the value matrix that have second scores higher than the second score threshold.

### Claim 14 (depends on 9)

The system of claim 9,
wherein the key cache has a cache size defined by dimensions B×S×L×N×D, where B is a batch size of the input sequence, S is a sequence length of the input sequence, L is a total number of layer in the Transformer-based neural network model, N is a number of heads in each layer, and D is a total number of channels in each head,
wherein the removing of the at least one or more rows of the key matrix prunes one or more channels from the dimension D,
further comprising performing cache eviction and/or structured pruning techniques to prune the key cache from the dimension S or the dimension L.

### Claim 15

A non-transitory machine-readable medium comprising a plurality of machine-executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:
transforming, by a Transformer-based neural network model, an input sequence of tokens into intermediate variables including at least a key matrix, a value matrix, and a query matrix;
allocating, in one or more processor memories, a key cache for storing the key matrix and a value cache for storing the value matrix;
generating a plurality of scores indicating magnitudes associated with a plurality of rows of the key matrix stored at the key cache, respectively;
re-allocating at least a portion of the key cache by removing at least one or more rows of the key matrix having associated scores below a score threshold, each row of the key matrix corresponding to a channel of the key cache; and
operating the Transformer-based neural network model with a reduced key cache after the re-allocating.

### Claim 16 (depends on 15)

The medium of claim 15, wherein the scores associated with each row in the key cache are attention scores computed using queries and keys in the respective query matrix and key matrix, wherein the re-allocating of the key cache filters the key matrix by only keeping rows in the key matrix that have attention scores higher than an attention score threshold.

### Claim 17 (depends on 16)

The medium of claim 16, wherein the key matrix is filtered through a channel mask matrix also stored in the key cache.

### Claim 18 (depends on 16)

The medium of claim 16, wherein the re-allocating further includes performing low-rank approximation on the key matrix based on the attention scores.

### Claim 19 (depends on 15)

The medium of claim 15, wherein the scores associated with each row in the key cache are absolute magnitude values of keys in the key matrix, wherein the re-allocating of the key cache filters the key matrix by only keeping channels in the rows that have magnitude values higher than a magnitude score threshold.

### Claim 20 (depends on 15)

The medium of claim 15,
wherein the key cache has a cache size defined by dimensions B×S×L×N×D, where B is a batch size of the input sequence, S is a sequence length of the input sequence, L is a total number of layer in the Transformer-based neural network model, N is a number of heads in each layer, and D is a total number of channels in each head,
wherein the removing of the at least one or more rows of the key matrix prunes one or more channels from the dimension D,
further comprising performing cache eviction and/or structured pruning techniques to prune the key cache from the dimension S or the dimension L.
