# US 2020/0401529 A1 · Gpu cache management based on locality type detection

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0401529 A1
- Document ID: US20200401529A1
- Status: Published application
- Kind: A1
- Application number: 16/446,119
- Filed: 2019-06-19
- Published: 2020-12-24
- Earliest claimed: 2019-06-19
- Assignees: Advanced Micro Devices, Inc.
- Applicants: Advanced Micro Devices, Inc.
- Inventors: Xianwei ZHANG; John KALAMATIANOS; Bradford BECKMANN
- Web page: https://patentgrep.com/patent/US20200401529A1

## CPC

- G06F 12/0891: using clearing, invalidating or resetting means
- G06F 9/3877: {using a secondary processor, e.g. coprocessor (peripheral processor G06F13/12)}
- G06F 9/5016: {the resource being the memory}
- G06F 9/5066: {Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451)}
- G06F 9/545: {where tasks reside in different layers, e.g. user- and kernel-space}
- G06F 12/0888: using selective caching, e.g. bypass
- G06F 12/0895: of parts of caches, e.g. directory or tag array
- G06F 12/126: with special data handling, e.g. priority of data or instructions, handling errors or pinning
- G06F 2212/1021: Hit rate improvement
- G06F 2212/302: In image processor or graphics adapter
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Wavefront loading in a processor is managed and includes monitoring a selected wavefront of a set of wavefronts. Reuse of memory access requests for the selected wavefront is counted. A cache hit rate in one or more caches of the processor is determined based on the counted reuse. Based on the cache hit rate, subsequent memory requests of other wavefronts of the set of wavefronts are modified by including a type of reuse of cache lines in requests to the caches. In the caches, storage of data in the caches is based on the type of reuse indicated by the subsequent memory access requests. Reused cache lines are protected by preventing cache line contents from being replaced by another cache line for a duration of processing the set of wavefronts. Caches are bypassed when streaming access requests are made.

## Claims

### Claim 1

A method for managing wavefront loading in a processor, the method comprising:
monitoring a selected wavefront of a set of wavefronts, the selected wavefront executed by one or more compute units of a group of compute units of the processor by:
counting reuse of memory access requests for the selected wavefront; and
calculating a cache hit rate of hits in one or more caches of the processor for instructions for the selected wavefront based on the counted reuse of the memory access requests; and
based on the calculated cache hit rate, modifying subsequent memory requests of other wavefronts of the set of wavefronts by including a type of cache reuse in requests to at least one cache in the processor, wherein the type of cache reuse is based on an amount of cache reuse relative to a cache reuse threshold value.

### Claim 2 (depends on 1)

The method of claim 1, wherein monitoring occurs once per wavefront kernel launch.

### Claim 3 (depends on 1)

The method of claim 1, wherein monitoring includes monitoring load instructions of the selected wavefront across a same group of compute units and aggregating hits in the one or more caches of the compute units of the same group for calculating the cache hit rate.

### Claim 4 (depends on 1)

The method of claim 1, wherein counting reuse of the memory access requests includes:
inserting a tag entry into a monitor array of a compute unit, the monitor array having tag entries for tracking memory access requests to the compute unit within the selected wavefront; and
updating a count of a number of reuses of data in a reuse field of an entry of a locality detection table, the locality detection table having a set of entries, each entry including the reuse field.

### Claim 5 (depends on 4)

The method of claim 4, wherein the monitor array is direct-mapped to addresses of a cache of the group of compute units of the processor.

### Claim 6 (depends on 4)

The method of claim 4, wherein each tag entry of the monitor array includes a part of a memory access request target address, and wherein the method further comprises:
managing storage of data at the one or more caches based on the type of reuse of cache lines indicated by the subsequent memory access requests by:
protecting a reused cache line by preventing cache line contents from being replaced by another cache line for a duration of processing the set of wavefronts; or
bypassing a cache for streaming access requests for potential victim cache entries for the duration of processing the set of wavefronts.

### Claim 7 (depends on 4)

The method of claim 4, wherein each entry of the locality detection table further includes a no-reuse field, the method further comprising:
updating a count of a number of no-reuses of data in the no-reuse field made by the wavefront memory access requests for data that are loaded in a cache and are not reused by the monitored wavefront.

### Claim 8 (depends on 4)

The method of claim 4, wherein monitoring includes:
inserting a tag entry of the monitor array into a tag entry in a Bloom filter coupled to the monitor array when the monitor array is full of entries and overflows, the Bloom filter having its own a set of entries.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
based on the calculated cache hit rate, setting a reuse bit of subsequent cache requests associated with the set of wavefronts, the reuse bit indicating high temporal locality with a relatively high amount of data reuse in one or more caches of the group of compute units of the processor.

### Claim 10 (depends on 1)

The method of claim 1, wherein each compute unit of the group of compute units includes a coalescer, the method further comprising:
consolidating the memory access requests by the coalescer, and wherein monitoring includes counting the reuse of memory access requests issued from the respective coalescers.

### Claim 11

A processor comprising:
a first memory cache; and
a first compute unit of a group of parallel processing compute units, each compute unit including:
a monitor array having tag entries for identifying memory access requests within a wavefront of a set of wavefronts loaded into the processor for execution; and
a locality detection table having a set of entries corresponding to a respective program counter of the memory access requests of the wavefront, wherein each entry of the locality detection table includes:
a reuse field for a count of a number of reuses of data made by the wavefront memory access requests; and
a status field for the memory access requests.

### Claim 12 (depends on 11)

The processor of claim 11, wherein each entry of the locality detection table further includes a no-reuse field to count streaming vector loads in the processor.

### Claim 13 (depends on 11)

The processor of claim 11, further comprising:
the group of compute units including the first compute unit and a second compute unit for processing wavefronts of the set of wavefronts; and
a second memory cache shared among the compute units of the group of compute units.

### Claim 14 (depends on 11)

The processor of claim 11, wherein each compute unit of the group of parallel processing compute units further includes a coalescer configured to consolidate memory cache requests to the first memory cache.

### Claim 15 (depends on 11)

The processor of claim 11, wherein the first compute unit further includes a Bloom filter coupled to the monitor array and having a set of tag entries, the Bloom filter configured to receive and store overflow entries evicted from the monitor array.

### Claim 16 (depends on 11)

The processor of claim 11, further comprising:
a memory controller having a scheduler, the scheduler providing the wavefronts of the set of wavefronts to the group of compute units for execution by the processor as part of a parallel execution pipeline.

### Claim 17

A processing system comprising:
a central processing unit (CPU); and
a group of parallel processing compute units coupled to the CPU and having memory caches, the CPU providing a kernel for the group of parallel processing compute units to execute, the group of parallel processing compute units comprising at least one compute unit that includes:
a monitor array having tag entries for identifying memory access requests within a wavefront of a set of wavefronts loaded into the group of parallel processing compute units for execution;
a locality detection table having a set of entries corresponding to a respective program counter of the memory access requests of the wavefront, wherein each entry of the locality detection table includes:
a reuse field for a count of a number of reuses of data made by the wavefront memory access requests; and
a coalescer configured to consolidate memory access requests to the memory caches.

### Claim 18 (depends on 17)

The processing system of claim 17, wherein each entry of the locality detection table further includes a no-reuse field to count streaming vector loads of the wavefront of the set of wavefronts as the wavefront is processed by the group of parallel processing compute units in the processing system.

### Claim 19 (depends on 17)

The processing system of claim 17, wherein the at least one compute unit further includes:
a Bloom filter coupled to the monitor array, the Bloom filter having its own set of entries and being configured to receive an entry from the monitor array when the monitor array is full of entries and evicts an entry.

### Claim 20 (depends on 17)

The processing system of claim 17, wherein each of the memory caches includes:
an installation bypass module coupled to the locality detection table and configured to prevent a cache line of the memory cache from receiving data based on the set of entries of the locality detection table; or
a replacement protection module coupled to the locality detection table and configured to protect cache lines of the memory cache from eviction based on the set of entries of the locality detection table; or
both an installation bypass structure and a replacement protection module.