# US 12,112,397 B2 · Programmable coarse grained and sparse matrix compute hardware with advanced scheduling

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,397 B2
- Document ID: US12112397B2
- Status: Granted
- Kind: B2
- Application number: 18/334,733
- Filed: 2023-06-14
- Published: 2023-12-07
- Granted: 2024-10-08
- Earliest claimed: 2017-04-28
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Eriko Nurvitadhi; Balaji Vembu; Nicolas C. Galoppo Von Borries; Rajkishore Barik; Tsung-Han Lin; Kamal Sinha; Nadathur Rajagopalan Satish; Jeremy Bottleson; Farshad Akhbari; Altug Koker; Narayan Srinivasa; Dukhwan Kim; Sara S. Baghsorkhi; Justin E. Gottschlich; Feng Chen; Elmoustapha Ould-Ahmed-Vall; Kevin Nealis; Xiaoming Chen; Anbang Yao
- Web page: https://patentgrep.com/patent/US12112397B2

## CPC

- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06F 9/3001: {Arithmetic instructions}
- G06F 9/3017: {Runtime instruction translation, e.g. macros}
- G06F 9/30196: {using decoder, e.g. decoder per instruction set, adaptable or programmable decoders}
- G06F 9/3851: {from multiple instruction streams, e.g. multistreaming}
- G06F 9/3887: {controlled by a single instruction for multiple data lanes [SIMD]}
- G06F 9/3888: {controlled by a single instruction for multiple threads [SIMT] in parallel}
- G06F 9/38885: {Divergence aspects}
- G06F 9/3895: {for complex operations, e.g. multidimensional or interleaved address generators, macros}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/098: Distributed learning, e.g. federated learning
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

One embodiment provides a parallel processor comprising a hardware scheduler to schedule pipeline commands for compute operations to one or more of multiple types of compute units, a plurality of processing resources including a first sparse compute unit configured for input at a first level of sparsity and hybrid memory circuitry including a memory controller, a memory interface, and a second sparse compute unit configured for input at a second level of sparsity that is greater than the first level of sparsity.

## Claims

### Claim 1

A parallel processor comprising:
a hardware scheduler to schedule pipeline commands for compute operations to one or more of multiple types of compute units, wherein the multiple types of compute units include a first sparse compute unit configured for input at a first level of sparsity and a second sparse compute unit configured for input at a second level of sparsity that is greater than the first level of sparsity;
a plurality of processing resources coupled with the hardware scheduler, the plurality of processing resources including the first sparse compute unit; and
hybrid memory circuitry coupled with the plurality of processing resources, the hybrid memory circuitry including a memory controller, a memory interface, and the second sparse compute unit.

### Claim 2 (depends on 1)

The parallel processor of claim 1, the plurality of processing resources configured to access a device memory of the parallel processor via the memory interface of the hybrid memory circuitry.

### Claim 3 (depends on 1)

The parallel processor of claim 1, the first sparse compute unit and the second sparse compute unit each include an array of processing elements configured to perform matrix operations on a plurality of input matrices.

### Claim 4 (depends on 3)

The parallel processor of claim 3, the first sparse compute unit including a first number of processing elements and the second sparse compute unit including a second number of processing elements that is less than the first number of processing elements.

### Claim 5 (depends on 4)

The parallel processor of claim 4, the first sparse compute unit including an internal randomly accessible memory and circuitry configured to store non-zero value elements of a first input matrix having the first level of sparsity to the internal randomly accessible memory.

### Claim 6 (depends on 4)

The parallel processor of claim 4, the second sparse compute unit including a cache memory and second circuitry, the second circuitry configured to perform a gather operation to gather non-zero value elements of an input matrix having the second level of sparsity and store the non-zero value elements to the cache memory.

### Claim 7 (depends on 6)

The parallel processor of claim 6, the second circuitry configured to gather the non-zero value elements from a second randomly accessible memory via the memory interface, the second randomly accessible memory including device memory of the parallel processor.

### Claim 8 (depends on 1)

The parallel processor of claim 1, further comprising a microcontroller configured to implement the hardware scheduler.

### Claim 9 (depends on 8)

The parallel processor of claim 8, the hardware scheduler configured to:
receive a first instruction to be scheduled;
schedule a first plurality of pipeline commands to the first sparse compute unit to implement operations associated with the first instruction;
receive a second instruction to be scheduled; and
schedule a second plurality of pipeline commands to the second sparse compute unit to implement operations associated with the second instruction.

### Claim 10

A data processing system comprising:
a parallel processor comprising:
a hardware scheduler to schedule pipeline commands for compute operations to one or more of multiple types of compute units, wherein the multiple types of compute units include a first sparse compute unit configured for input at a first level of sparsity and a second sparse compute unit configured for input at a second level of sparsity that is greater than the first level of sparsity;
a plurality of processing resources coupled with the hardware scheduler, the plurality of processing resources including the first sparse compute unit; and
hybrid memory circuitry coupled with the plurality of processing resources, the hybrid memory circuitry including a memory controller, a memory interface, and the second sparse compute unit; and
a memory device coupled with the memory interface of the hybrid memory circuitry.

### Claim 11 (depends on 10)

The data processing system of claim 10, the plurality of processing resources configured to access the memory device via the memory interface of the hybrid memory circuitry, wherein the memory device is a device memory of the data processing system.

### Claim 12 (depends on 10)

The data processing system of claim 10, the first sparse compute unit and the second sparse compute unit each include an array of processing elements configured to perform matrix operations on a plurality of input matrices.

### Claim 13 (depends on 12)

The data processing system of claim 12, the first sparse compute unit including a first number of processing elements and the second sparse compute unit including a second number of processing elements that is less than the first number of processing elements.

### Claim 14 (depends on 10)

The data processing system of claim 10, further comprising a microcontroller configured to execute firmware to implement the hardware scheduler, wherein the multiple types of compute units additionally include general-purpose graphics compute units and the plurality of processing resources includes a general-purpose graphics compute unit.

### Claim 15 (depends on 14)

The data processing system of claim 14, the hardware scheduler to schedule a pipeline command to the general-purpose graphics compute unit to cause execution of instructions associated with a kernel program.

### Claim 16 (depends on 15)

The data processing system of claim 15, the hybrid memory circuitry including a near-data compute unit and the kernel program includes an instruction to offload a near-memory compute operation to the near-data compute unit.

### Claim 17

A method comprising:
receiving a first instruction for execution at a parallel processor, the parallel processor including a first sparse compute unit configured for input at a first level of sparsity and a second sparse compute unit configured for input at a second level of sparsity that is greater than the first level of sparsity;
scheduling pipeline commands to enable execution of the first instruction via a hardware scheduler of the parallel processor, the pipeline commands scheduled to the first sparse compute unit;
receiving a second instruction for execution at the parallel processor; and
scheduling pipeline commands to enable execution of the second instruction via the hardware scheduler of the parallel processor, the pipeline commands scheduled to the second sparse compute unit.

### Claim 18 (depends on 17)

The method of claim 17, wherein the first instruction is determined to be executed by the first sparse compute unit and the second instruction is determined to be executed by the second sparse compute unit.

### Claim 19 (depends on 18)

The method of claim 18, wherein the first instruction is pre-determined to be executed by the first sparse compute unit by an instruction set architecture of the parallel processor.

### Claim 20 (depends on 18)

The method of claim 18, wherein the second instruction is determined to be executed by the second sparse compute unit based on analysis of an input matrix associated with the second instruction.