# US 2021/0240524 A1 · Methods and apparatus to facilitate tile-based gpu machine learning acceleration

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240524 A1
- Document ID: US20210240524A1
- Status: Published application
- Kind: A1
- Application number: 16/779,275
- Filed: 2020-01-31
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Hitendra Mohan GANGANI; Balaji CALIDAS; Murat BALCI
- Web page: https://patentgrep.com/patent/US20210240524A1

## CPC

- G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues}
- G06F 9/5061: {Partitioning or combining of resources}
- G06F 9/5016: {the resource being the memory}
- G06N 20/00: Machine learning
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 1/60: Memory management
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The present disclosure relates to methods and apparatus for machine learning processing. For example, disclosed techniques facilitate tile-based GPU machine learning acceleration. Aspects of the present disclosure can determine a tile size based on a memory size of a first memory and a job input size associated with executing a computational job. In some examples, the computational job may be one of a quantity of computational jobs configured to execute a machine learning primitive. Aspects of the present disclosure can also load, based on the tile size, input data associated with a batch of computational jobs from a second memory to the first memory. Further, aspects of the present disclosure can generate batch output data by executing the batch of computational jobs using the input data loaded to the first memory. Additionally, aspects of the present disclosure can store the generated batch output data to the second memory.

## Claims

### Claim 1

A method of machine learning processing, comprising:
determining a tile size based on a memory size of a first memory and a job input size associated with executing a computational job, the computational job being one of a quantity of computational jobs configured to execute a machine learning primitive;
loading, based on the tile size, input data associated with a batch of computational jobs from a second memory to the first memory;
generating batch output data by executing the batch of computational jobs using the input data loaded to the first memory; and
storing the generated batch output data to the second memory.

### Claim 2 (depends on 1)

The method of claim 1, wherein the first memory is associated with a first latency, and the second memory is associated with a second latency that is greater than the first latency.

### Claim 3 (depends on 1)

The method of claim 1, wherein the job input size associated with executing the computational job is determined based on a memory size of input data used to execute the computational job.

### Claim 4 (depends on 1)

The method of claim 1, wherein the determining of the tile size is further based on a job output size associated with executing the computational job, and wherein the job output size is determined based on a memory size of output data generated by the execution of the computational job.

### Claim 5 (depends on 4)

The method of claim 4, wherein the storing of the generated batch output data to the second memory further comprises:
writing the output data generated by the execution of each computational job of the batch of computational jobs to the first memory; and
storing the generated output data from the first memory to the second memory after execution of the batch of computational jobs is complete.

### Claim 6 (depends on 5)

The method of claim 5, wherein the batch of computational jobs is a first batch of computational jobs, and further comprising loading input data associated with a second batch of computational jobs from the second memory to the first memory, the loading of the input data associated with the second batch of computational jobs being performed in parallel with the storing of the generated output data to the second memory after execution of the first batch of computational jobs is complete.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
loading second input data associated with executing a second batch of computational jobs from the second memory to the first memory;
generating second batch output data by executing the second batch of computational jobs using the second input data loaded to the first memory; and
storing the generated second batch output data to the second memory.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first memory is an on-chip memory of a graphics processor.

### Claim 9 (depends on 8)

The method of claim 8, wherein the second memory is accessible to the graphics processor and to a central processor.

### Claim 10 (depends on 8)

The method of claim 8, wherein the graphics processor comprises a plurality of processing elements configured to execute the batch of computational jobs.

### Claim 11 (depends on 1)

The method of claim 1, wherein the tile size corresponds to a quantity of computational jobs of the batch of computational jobs.

### Claim 12

An apparatus for machine learning processing, comprising:
a memory; and
at least one processor coupled to the memory and configured to:
determine a tile size based on a memory size of a first memory and a job input size associated with executing a computational job, the computational job being one of a quantity of computational jobs configured to execute a machine learning primitive;
load, based on the tile size, input data associated with a batch of computational jobs from a second memory to the first memory;
generate batch output data by executing the batch of computational jobs using the input data loaded to the first memory; and
store the generated batch output data to the second memory.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the first memory is associated with a first latency, and the second memory is associated with a second latency that is greater than the first latency.

### Claim 14 (depends on 12)

The apparatus of claim 12, wherein the job input size associated with executing the computational job is determined based on a memory size of input data used to execute the computational job.

### Claim 15 (depends on 12)

The apparatus of claim 12, wherein the at least one processor is configured to determine the tile size based on a job output size associated with executing the computational job, the job output size being determined based on a memory size of output data generated by the execution of the computational job.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the at least one processor is configured to store the generated batch output data to the second memory by:
writing the output data generated by the execution of each computational job of the batch of computational jobs to the first memory; and
storing the generated output data from the first memory to the second memory after execution of the batch of computational jobs is complete.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the batch of computational jobs is a first batch of computational jobs, and the at least one processor is configured to load input data associated with a second batch of computational jobs from the second memory to the first memory, the loading of the input data associated with the second batch of computational jobs being performed in parallel with the storing of the generated output data to the second memory after execution of the first batch of computational jobs is complete.

### Claim 18 (depends on 12)

The apparatus of claim 12, wherein the at least one processor is further configured to:
load second input data associated with executing a second batch of computational jobs from the second memory to the first memory;
generate second batch output data by executing the second batch of computational jobs using the second input data loaded to the first memory; and
store the generated second batch output data to the second memory.

### Claim 19 (depends on 12)

The apparatus of claim 12, wherein the first memory is an on-chip memory of a graphics processor.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the second memory is accessible to the graphics processor and to a central processor.

### Claim 21 (depends on 12)

The apparatus of claim 12, wherein the at least one processor comprises a plurality of processing elements configured to execute the batch of computational jobs.

### Claim 22 (depends on 12)

The apparatus of claim 12, wherein the tile size corresponds to a quantity of computational jobs of the batch of computational jobs.

### Claim 23 (depends on 12)

The apparatus of claim 12, wherein the apparatus includes a wireless communication device.

### Claim 24

A non-transitory computer-readable medium storing computer executable code for machine learning processing, comprising code to:
determine a tile size based on a memory size of a first memory and a job input size associated with executing a computational job, the computational job being one of a quantity of computational jobs configured to execute a machine learning primitive;
load, based on the tile size, input data associated with a batch of computational jobs from a second memory to the first memory;
generate batch output data by executing the batch of computational jobs using the input data loaded to the first memory; and
store the generated batch output data to the second memory.
