# US 2021/0334648 A1 · Statically Generated Compiled Representations for Processing Data in Neural Networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334648 A1
- Document ID: US20210334648A1
- Status: Published application
- Kind: A1
- Application number: 16/894,025
- Filed: 2020-06-05
- Published: 2021-10-28
- Earliest claimed: 2020-04-28
- Assignees: Advanced Micro Devices, Inc.
- Applicants: Advanced Micro Devices, Inc.
- Inventors: Xiuyu Li; Jian Yang
- Web page: https://patentgrep.com/patent/US20210334648A1

## CPC

- G06N 3/067: using optical means
- G06N 3/08: Learning methods
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0499: Feedforward networks
- G06N 3/09: Supervised learning
- G06N 3/105: {Shells for specifying net layout}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An electronic device includes a memory that stores input matrices A and B, a cache memory, and a processor. The processor generates a compiled representation that includes values for acquiring data from input matrix A when processing instances of input data through the neural network, the values including a base address in input matrix A for each thread from among a number of threads and relative offsets, the relative offsets being distances between elements of input matrix A to be processed by the threads. The processor then stores, in the local cache memory, the compiled representation including the base address for each thread and the relative offsets.

## Claims

### Claim 1

An electronic device, comprising:
a local cache memory;
a memory, the memory storing input matrix A, input matrix A having values to be used when processing instances of input data through a neural network; and
a processor, the processor configured to:
generate a compiled representation that includes values for acquiring data from input matrix A when processing instances of input data through the neural network, the values including a base address in input matrix A for each thread from among a number of threads and relative offsets, the relative offsets being distances between elements of input matrix A to be processed by the threads; and
store, in the local cache memory, the compiled representation including the base address for each thread and the relative offsets.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein, when generating the compiled representation, the processor is configured to:
compute the base address for each thread in input matrix A as a function of some or all of a thread identifier (ID) for that thread, dimensions of input matrix A and/or an output matrix C, properties of elements of input matrix A and/or output matrix C, and convolutional filter properties.

### Claim 3 (depends on 1)

The electronic device of claim 1, wherein, when generating the compiled representation, the processor is configured to:
compute the relative offsets as a function of some or all of dimensions of input matrix A and/or an output matrix C, properties of elements of input matrix A and/or output matrix C, and filter properties.

### Claim 4 (depends on 1)

The electronic device of claim 1, wherein:
the memory stores input matrix B, input matrix B having values to be used when processing instances of input data through a neural network; and
the processor is further configured to:
process input matrix A using each of the threads, the processing including using the compiled representation in the local cache memory to identify locations in memory from which values in elements of input matrix A are to be acquired to be used, along with values from elements at corresponding locations from input matrix B, as inputs for one or more general matrix multiplication (GEMM) operations by that thread.

### Claim 5 (depends on 4)

The electronic device of claim 4, wherein, when processing input matrix A using each of the threads, the processor is configured to:
acquire, from the compiled representation for input matrix A in the local cache memory, the base address for each thread for input matrix A;
while elements of input matrix A remain to be processed, for an active thread from among the threads:
acquire, from the compiled representation, a next relative offset;
increase that active thread's base address by the next relative offset;
read a first value from an element of input matrix A at a location in the memory identified by that active thread's base address;
read a second value from an element of input matrix B at a memory location associated with the element of input matrix A;
use, as inputs to a GEMM operation, the first value and the second value; and
store, in an output matrix C in the memory, a result of the GEMM operation.

### Claim 6 (depends on 5)

The electronic device of claim 5, wherein the processor is further configured to use an index for acquiring the relative offsets in combination with a base address for input matrix B for reading the second value from input matrix B.

### Claim 7 (depends on 5)

The electronic device of claim 5, wherein, when storing, in output matrix C, the results of the GEMM operation, the processor is further configured to:
store each result in output matrix C at a location in output matrix C that is determined based on a location in input matrix A from where the first value was read.

### Claim 8 (depends on 4)

The electronic device of claim 4, wherein the values in input matrix A and input matrix B include input values and weights, respectively, associated with instances of input data to be used for processing the instances of input data through the neural network.

### Claim 9 (depends on 4)

The electronic device of claim 4, wherein the dimensions of input matrix A and input matrix B are greater than dimensions used for the GEMM operations.

### Claim 10 (depends on 1)

The electronic device of claim 1, wherein the local cache memory is coupled to the processing circuitry via a fast-access interface that enables faster accesses than accesses of the memory.

### Claim 11

A method for generating a compiled representation in an electronic device that includes a local cache memory; a memory, the memory storing an input matrix A, input matrix A having values to be used when processing instances of input data through the neural network; and a processor, the method comprising:
generating, by the processor, a compiled representation that includes values for acquiring data from input matrix A when processing instances of input data through the neural network, the values including a base address in input matrix A for each thread from among a number of threads and relative offsets, the relative offsets being distances between elements of input matrix A to be processed by the threads; and
storing, by the processor, in the local cache memory, the compiled representation including the base address for each thread and the relative offsets.

### Claim 12 (depends on 11)

The method of claim 11, wherein generating the compiled representation comprises:
computing, by the processor, the base address for each thread in input matrix A as a function of some or all of a thread identifier (ID) for that thread, dimensions of input matrix A and/or an output matrix C, properties of elements of input matrix A and/or output matrix C, and convolutional filter properties.

### Claim 13 (depends on 11)

The method of claim 11, wherein generating the compiled representation comprises:
computing, by the processor, the relative offsets for input matrix A as a function of some or all of dimensions of input matrix A and/or an output matrix C, properties of elements of input matrix A and/or output matrix C, and filter properties.

### Claim 14 (depends on 11)

The method of claim 11, wherein:
the memory stores an input matrix B, input matrix B having values to be used when processing instances of input data through a neural network; and
the method further comprises:
processing, by the processor, input matrix A using each of the threads, the processing including using the compiled representation in the local cache memory to identify locations in memory from which values in elements of input matrix A are to be acquired to be used, along with values from elements at corresponding locations from input matrix B, as inputs for one or more general matrix multiplication (GEMM) operations by that thread.

### Claim 15 (depends on 14)

The method of claim 14, wherein processing input matrix A using each of the threads includes:
acquiring, by the processor, from the compiled representation for input matrix A in the local cache memory, the base address for each thread for input matrix A and the relative offsets;
while elements of input matrix A remain to be processed, for an active thread from among the threads:
read, by the processor, a first value from an element of input matrix A at a location in the memory identified by that active thread's base address for input matrix A as adjusted by a corresponding relative offset;
read, by the processor, a second value from an element of input matrix B at a memory location associated with the element of input matrix A;
use, by the processor, as inputs to a GEMM operation, the first value and the second value; and
store, by the processor, in an output matrix C in the memory, a result of the GEMM operation.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
using, by the processor, an index for acquiring the relative offsets in combination with a base address for input matrix B for reading the second value from input matrix B.

### Claim 17 (depends on 15)

The method of claim 15, wherein storing, in output matrix C, the results of the GEMM operation includes:
storing, by the processor, each result in output matrix C at a location in output matrix C that is determined based on a location in input matrix A from where the first value was read.

### Claim 18 (depends on 14)

The method of claim 14, wherein the values in input matrix A and input matrix B include input values and weights, respectively, associated with instances of input data to be used for processing the instances of input data through the neural network.

### Claim 19 (depends on 14)

The method of claim 14, wherein the dimensions of input matrix A and input matrix B are greater than dimensions used for the GEMM operations.

### Claim 20

An electronic device, comprising:
a memory storing input matrix A and input matrix B, input matrix A and input matrix B having respective values to be used when processing instances of input data through a neural network;
a local cache memory that stores a compiled representation, the compiled representation including values for acquiring data from input matrix A when processing instances of input data through the neural network, the values including a base address in input matrix A for each thread from among a number of threads and relative offsets, the relative offsets being distances between elements of input matrix A to be processed by the threads; and
a processor configured to:
acquire, from the local cache memory, the compiled representation; and
process input matrix A by, for each thread from among the number of threads, using the compiled representation to identify locations in memory from which values in elements of input matrix A are to be acquired to be used, along with values from elements at corresponding locations from input matrix B, as inputs for one or more general matrix multiplication (GEMM) operations by that thread.

### Claim 21 (depends on 20)

The electronic device of claim 20, wherein, when processing input matrix A, the processor is configured to:
acquire, from the compiled representation for input matrix A in the local cache memory, the base address for each thread for input matrix A;
while elements of input matrix A remain to be processed, for an active thread from among the threads:
acquire, from the compiled representation, a next relative offset;
increase that active thread's base address by the next relative offset;
read a first value from an element of input matrix A at a location in the memory identified by that active thread's base address;
read a second value from an element of input matrix B at a memory location associated with the element of input matrix A;
use, as inputs to a GEMM operation, the first value and the second value; and
store, in an output matrix C in the memory, a result of the GEMM operation.

### Claim 22 (depends on 21)

The electronic device of claim 21, wherein the processor is further configured to use an index for acquiring the relative offsets in combination with a base address for input matrix B for reading the second value from input matrix B.

### Claim 23 (depends on 21)

The electronic device of claim 21, wherein, when storing, in output matrix C, the results of the GEMM operation, the processor is further configured to:
store each result in output matrix C at a location in output matrix C that is determined based on a location in input matrix A from where the first value was read.

### Claim 24

A method for processing instances of input data through a neural network in an electronic device that includes a local cache memory that stores a compiled representation; a memory storing input matrix A and input matrix B, input matrix A and input matrix B having respective values to be used when processing instances of input data through a neural network; and a processor, the method comprising:
acquiring, by the processor, from the local cache memory, the compiled representation, the compiled representation including values for acquiring data from input matrix A when processing instances of input data through the neural network, the values including a base address in input matrix A for each thread from among a number of threads and relative offsets, the relative offsets being distances between elements of input matrix A to be processed by the threads; and
processing, by the processor, input matrix A by, for each thread from among the number of threads, using the compiled representation to identify locations in memory from which values in elements of input matrix A are to be acquired to be used, along with values from elements at corresponding locations from input matrix B, as inputs for one or more general matrix multiplication (GEMM) operations by that thread.

### Claim 25 (depends on 24)

The method of claim 24, wherein processing input matrix A comprises:
acquiring, by the processor, from the compiled representation for input matrix A in the local cache memory, the base address for each thread for input matrix A;
while elements of input matrix A remain to be processed, for an active thread from among the threads:
acquiring, by the processor, from the compiled representation, a next relative offset;
increasing, by the processor, that active thread's base address by the next relative offset;
reading, by the processor, a first value from an element of input matrix A at a location in the memory identified by that active thread's base address;
reading, by the processor, a second value from an element of input matrix B at a memory location associated with the element of input matrix A;
using, by the processor, as inputs to a GEMM operation, the first value and the second value; and
storing, by the processor, in an output matrix C in the memory, a result of the GEMM operation

### Claim 26 (depends on 25)

The method of claim 25, further comprising using, by the processor, an index for acquiring the relative offsets in combination with a base address for input matrix B for reading the second value from input matrix B.

### Claim 27 (depends on 25)

The method of claim 25, wherein, storing, in output matrix C, the results of the GEMM operation, comprises:
storing, by the processor, each result in output matrix C at a location in output matrix C that is determined based on a location in input matrix A from where the first value was read.