# US 2025/0156326 A1 · Managing a system level cache

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0156326 A1
- Document ID: US20250156326A1
- Status: Published application
- Kind: A1
- Application number: 18/505,952
- Filed: 2023-11-09
- Published: 2025-05-15
- Earliest claimed: 2023-11-09
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Ashutosh PANDEY; Steven E. MOLNAR; Mirko ANDJIC; Amit PABALKAR; Harold W. CAIN, III; Prabal SHARMA
- Web page: https://patentgrep.com/patent/US20250156326A1

## CPC

- G06F 12/0806: Multiuser, multiprocessor or multiprocessing cache systems
- G06F 12/0811: with multilevel cache hierarchies
- G06F 12/0848: {Partitioned cache, e.g. separate instruction and operand caches}
- G06F 12/126: with special data handling, e.g. priority of data or instructions, handling errors or pinning
- G06F 12/128: adapted to multidimensional cache systems, e.g. set-associative, multicache, multiset or multilevel
- G06F 2212/1044: Space efficiency improvement
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Various embodiments include techniques for managing cache memory in a computing system. The disclosed techniques include a cache policy manager that monitors activity of various components that access a common cache memory. The cache policy manager establishes cache rules that determine what data remains stored in cache memory and what data is removed from cache memory in order to make room for new data. As the activities of these components change over time, cache rules that work well for a previous activity profile may no longer work well for the current activity profile. Therefore, the cache policy manager dynamically modifies the cache rules as the activity profile changes in order to select cache rules at any given time that work well with the current activity profile. These techniques are advantageous over conventional approaches that employ static cache rules that work well only for specific activity profiles.

## Claims

### Claim 1

A computer-implemented method for managing a cache memory in a computing system, the method comprising:
identifying a system state of the computing system;
determining a memory access pattern of a client of the cache memory that is executing in the computing system; and
determining a plurality of cache control parameters based on the system state and the memory access pattern.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the system state comprises an indication of whether the client of the cache memory is in an active state or in a low-power state.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the system state comprises an indication of whether the client of the cache memory has initiated a streaming operation or has ceased the streaming operation.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein determining the plurality of cache control parameters comprises changing a partition boundary to increase an amount of system level cache memory allocated to a first system level cache partition included in a plurality of system level cache memory partitions and to decrease an amount of system level cache memory allocated to a second system level cache partition included in the plurality of system level cache memory partitions.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the client of the cache memory is included in a plurality of clients of the cache memory, and wherein at least one of the first system level cache partition or the second system level cache partition is allocated to a subset of two or more clients of the cache memory included in the plurality of clients of the cache memory.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein determining the plurality of cache control parameters comprises changing an eviction policy of a cache memory based on the system state.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein determining the plurality of cache control parameters comprises:
identifying an allocation/replacement identifier (ARID) value associated with the client of the cache memory;
identifying a buffer type associated with the memory access pattern of the client of the cache memory;
identifying a table entry in a table of cache control parameters based on the ARID value and the buffer type; and
retrieving the plurality of cache control parameters from the table entry.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein determining the plurality of cache control parameters comprises:
identifying an allocation/replacement identifier (ARID) value associated with the client of the cache memory;
identifying a buffer type associated with the memory access pattern of the client of the cache memory;
identifying a first table entry in a first table of indices based on the ARID value and the buffer type;
retrieving an index from the first table entry;
identifying a second table entry in a second table of cache control parameters based on the index; and
retrieving the plurality of cache control parameters from the second table entry.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the system state includes an activity indication for one or more clients of the cache memory.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the system state includes power data or use case data determined by one or more of a power manager or an operating system.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein determining a memory access pattern of a client of the cache memory that is executing in the computing system comprises receiving attributes with each request from the client of the cache memory identifying a category of the request.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein determining a plurality of cache control parameters comprises indexing into a table of cache control parameters wherein the index is a function of a client identifier and a buffer type attribute.

### Claim 13 (depends on 12)

The computer-implemented method of claim 12, wherein contents of the table of cache control parameters are updated by a cache policy manager based on the system state.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein the plurality of cache control parameters is determined by at least one of a neural processing unit or a neural network processor based on by at least one of a client identifier, a buffer type attribute, or the system state.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, wherein the plurality of cache control parameters includes one or more of: a cache allocation policy or a cache eviction policy.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1, wherein the plurality of cache control parameters comprises at least one of global cache partition parameters, per-request allocation parameters, or per-request eviction parameters.

### Claim 17

A system, comprising:
a memory;
a plurality of cache memory clients that is coupled to the memory; and
a cache memory policy manager configured to:
identify a system state of the system;
identify a memory access pattern of a first cache memory client included in the plurality of cache memory clients that is executing in the system; and
determine a plurality of cache control parameters based on the system state and the memory access pattern.

### Claim 18 (depends on 17)

The system of claim 17, wherein the system state comprises an indication of whether the first cache memory client is in an active state or in a low-power state.

### Claim 19 (depends on 17)

The system of claim 17, wherein the system state comprises an indication of whether the first cache memory client has initiated a streaming operation or has ceased the streaming operation.

### Claim 20 (depends on 17)

The system of claim 17, wherein determining the plurality of cache control parameters comprises changing a partition boundary to increase an amount of system level cache memory allocated to a first system level cache partition included in a plurality of system level cache memory partitions and to decrease an amount of system level cache memory allocated to a second system level cache partition included in the plurality of system level cache memory partitions.
