# US 2024/0004718 A1 · Compiling tensor operators for neural network models based on tensor tile configurations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004718 A1
- Document ID: US20240004718A1
- Status: Published application
- Kind: A1
- Application number: 17/853,488
- Filed: 2022-06-29
- Published: 2024-01-04
- Earliest claimed: 2022-06-29
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Jilong XUE; Lingxiao MA; Yuqing XIA; Wei CUI; Fan YANG; Mao YANG; Lidong ZHOU
- Web page: https://patentgrep.com/patent/US20240004718A1

## CPC

- G06F 8/453: {Data distribution}
- G06F 9/5044: {considering hardware capabilities}
- G06F 8/443: {Optimisation}
- G06F 8/447: {Target code generation}
- G06F 9/5016: {the resource being the memory}
- G06F 9/5066: {Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs (mappping at compile time, see G06F8/451)}
- G06F 9/5072: {Grid computing}
- G06F 2209/5018: Thread allocation
- G06F 2209/509: Offload
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments of the present disclosure include systems and methods for compiling tensor operators for neural network models based on tensor tile configurations. A tensor expression for a neural network model is received. A tensor tile configuration for the tensor expression is determined based on specifications associated with a set of hardware devices. Based on the tensor tile configuration, a set of code for implementing the tensor expression for the neural network model is generated. The set of code is for execution on the set of hardware.

## Claims

### Claim 1

A non-transitory machine-readable medium storing a program executable by at least one processing unit of a device, the program comprising sets of instructions for:
receiving a tensor expression for a neural network model;
determining a tensor tile configuration for the tensor expression based on specifications associated with a set of hardware devices; and
based on the tensor tile configuration, generating a set of code for implementing the tensor expression for the neural network model, the set of code for execution on the set of hardware.

### Claim 2 (depends on 1)

The non-transitory machine-readable medium of claim 1, wherein the specifications associated with the set of hardware include a number of threads that can be processed concurrently by a particular hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of a shape of a tensor tile for a tensor tile configuration in the tensor tile configuration that is a multiple of the number of threads that can be processed concurrently by the particular hardware device.

### Claim 3 (depends on 1)

The non-transitory machine-readable medium of claim 1, wherein the specifications associated with the set of hardware include a length of a memory transaction utilized by a hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of an inner-most dimension of a tensor tile that is a multiple of the length of a memory transaction.

### Claim 4 (depends on 1)

The non-transitory machine-readable medium of claim 1, wherein the specifications associated with the set of hardware include a number of memory banks into which memory included in a hardware device is divided and a width of the memory banks, wherein the program further comprises instructions for determining an amount of padding for the tensor expression based on the number of memory banks and the width of the memory banks.

### Claim 5 (depends on 1)

The non-transitory machine-readable medium of claim 1, wherein the tensor expression specifying a set of shapes for a set of input tensors, a shape for an output tensor, and a set of tensor operations operating on the set of inputs tensors to generate the output tensor.

### Claim 6 (depends on 5)

The non-transitory machine-readable medium of claim 5, wherein determining the tensor tile configuration for the tensor expression comprises determining a shape of a tensor tile into which the shape of an input tensor in the set of input tensors can be evenly partitioned.

### Claim 7 (depends on 1)

The non-transitory machine-readable medium of claim 1, wherein determining the tensor tile configuration for the tensor expression comprises determining a tensor tile configuration for a particular hardware device in the set of hardware devices; and replicating the tensor tile configuration for each hardware device in the set of hardware devices other than the particular hardware device.

### Claim 8

A method comprising:
receiving a tensor expression for a neural network model;
determining a tensor tile configuration for the tensor expression based on specifications associated with a set of hardware devices; and
based on the tensor tile configuration, generating a set of code for implementing the tensor expression for the neural network model, the set of code for execution on the set of hardware.

### Claim 9 (depends on 8)

The method of claim 8, wherein the specifications associated with the set of hardware include a number of threads that can be processed concurrently by a particular hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of a shape of a tensor tile for a tensor tile configuration in the tensor tile configuration that is a multiple of the number of threads that can be processed concurrently by the particular hardware device.

### Claim 10 (depends on 8)

The method of claim 8, wherein the specifications associated with the set of hardware include a length of a memory transaction utilized by a hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of an inner-most dimension of a tensor tile that is a multiple of the length of a memory transaction.

### Claim 11 (depends on 8)

The method of claim 8, wherein the specifications associated with the set of hardware include a number of memory banks into which memory included in a hardware device is divided and a width of the memory banks, the method further comprising determining an amount of padding for the tensor expression based on the number of memory banks and the width of the memory banks.

### Claim 12 (depends on 8)

The method of claim 8, wherein the tensor expression specifying a set of shapes for a set of input tensors, a shape for an output tensor, and a set of tensor operations operating on the set of inputs tensors to generate the output tensor.

### Claim 13 (depends on 12)

The method of claim 12, wherein determining the tensor tile configuration for the tensor expression comprises determining a shape of a tensor tile into which the shape of an input tensor in the set of input tensors can be evenly partitioned.

### Claim 14 (depends on 8)

The method of claim 8, wherein determining the tensor tile configuration for the tensor expression comprises determining a tensor tile configuration for a particular hardware device in the set of hardware devices; and replicating the tensor tile configuration for each hardware device in the set of hardware devices other than the particular hardware device.

### Claim 15

A system comprising:
a set of processing units; and
a non-transitory machine-readable medium storing instructions that when executed by at least one processing unit in the set of processing units cause the at least one processing unit to:
receive a tensor expression for a neural network model;
determine a tensor tile configuration for the tensor expression based on specifications associated with a set of hardware devices; and
based on the tensor tile configuration, generate a set of code for implementing the tensor expression for the neural network model, the set of code for execution on the set of hardware.

### Claim 16 (depends on 15)

The system of claim 15, wherein the specifications associated with the set of hardware include a number of threads that can be processed concurrently by a particular hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of a shape of a tensor tile for a tensor tile configuration in the tensor tile configuration that is a multiple of the number of threads that can be processed concurrently by the particular hardware device.

### Claim 17 (depends on 15)

The system of claim 15, wherein the specifications associated with the set of hardware include a length of a memory transaction utilized by a hardware device in the set of hardware devices, wherein determining the tensor tile configuration for the tensor expression comprises determining a size of an inner-most dimension of a tensor tile that is a multiple of the length of a memory transaction.

### Claim 18 (depends on 15)

The system of claim 15, wherein the specifications associated with the set of hardware include a number of memory banks into which memory included in a hardware device is divided and a width of the memory banks, wherein the instructions further cause the at least one processing unit to determine an amount of padding for the tensor expression based on the number of memory banks and the width of the memory banks.

### Claim 19 (depends on 15)

The system of claim 15, wherein the tensor expression specifying a set of shapes for a set of input tensors, a shape for an output tensor, and a set of tensor operations operating on the set of inputs tensors to generate the output tensor.

### Claim 20 (depends on 19)

The system of claim 19, wherein determining the tensor tile configuration for the tensor expression comprises determining a shape of a tensor tile into which the shape of an input tensor in the set of input tensors can be evenly partitioned.
