Back to results
US 2026/0228135 A1Published application
Neural network cache eviction to avoid restore
AssigneeNVIDIA Corporation
InventorsJohn W. Thomson, Thor Johnsen, Kyle David Kranen
Life of this patent
Estimating the term…
Abstract
Apparatuses, systems, and techniques to identify information to evict from a Key-Value (KV) cache. In at least one embodiment, information stored within one or more large language model (LLM) KV caches may be identified to cause an indication to generated of information stored within the one or more LLM KV caches that can be evicted without causing other information stored within the one or more LLM KV caches to be restored to the one or more LLM KV caches.
Claims20
A processor, comprising:
- one or more circuits to cause an indication to be generated of information stored within one or more large language model (LLM) key-value (KV) caches that can be evicted from the one or more KV caches without causing other information stored within the one or more KV caches to be restored to the one or more KV caches.
A method, comprising:
- causing an indication to be generated of information stored within one or more large language model (LLM) key-value (KV) caches that can be evicted from the one or more KV caches without causing other information stored within the one or more KV caches to be restored to the one or more KV caches.
A system, comprising:
- one or more processors to cause an indication to be generated of information stored within one or more large language model (LLM) key-value (KV) caches that can be evicted from the one or more KV caches without causing other information stored within the one or more KV caches to be restored to the one or more KV caches;
- and
- one or more memories to store weights of one or more LLMs of the one or more LLM KV caches.
Dependencies are inferred from the claim text.