# US 2019/0318229 A1 · Method and system for hardware mapping inference pipelines

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318229 A1
- Document ID: US20190318229A1
- Status: Published application
- Kind: A1
- Application number: 15/952,131
- Filed: 2018-04-12
- Published: 2019-10-17
- Earliest claimed: 2018-04-12
- Assignees: Advanced Micro Devices, Inc.
- Applicants: Advanced Micro Devices, Inc.
- Inventors: Shuai Che
- Web page: https://patentgrep.com/patent/US20190318229A1

## CPC

- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/0499: Feedforward networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods and systems for hardware mapping inference pipelines in deep neural network (DNN) systems. Each layer of the inference pipeline is mapped to a queue, which in turn is associated with one or more processing elements. Each queue has multiple elements, where an element represents the task to be completed for a given input. Each input is associated with a queue packet which identifies, for example, a type of DNN layer, which DNN layer to use, a next DNN layer to use and a data pointer. A queue packet is written into the element of a queue, and the processing elements read the element and process the input based on the information in the queue packet. The processing element then writes another queue packet to another queue based on the processed queue packet. Multiple inputs can be processed in parallel and on-the-fly using the queues independent of layer starting points.

## Claims

### Claim 1

A deep neural network (DNN) system, comprising:
a plurality of queues;
a plurality of processing elements, wherein each queue of the plurality of queues is associated with at least one of the plurality of processing elements; and
an inference pipeline including a plurality of DNN layers, wherein each queue of the plurality of queues is mapped to one of the plurality of DNN layers,
wherein multiple inputs are processed in parallel by the plurality of queues and the plurality of processing elements, each queue and associated processing element being configured to process an input based on a DNN processing profile determined from a queue packet associated with the input.

### Claim 2 (depends on 1)

The DNN system of claim 1, wherein the queue packet identifies at least a DNN network identifier, a DNN layer identifier, a pointer to buffer for data, and previous/next DNN layer identifiers.

### Claim 3 (depends on 2)

The DNN system of claim 2, wherein the DNN layer identifier identifies a DNN layer type, which is used to determine a nature of computation to be performed and what kernels to launch.

### Claim 4 (depends on 2)

The DNN system of claim 2, wherein the DNN network identifier enables processing of multiple DNN workloads by designating which network to use.

### Claim 5 (depends on 2)

The DNN system of claim 2, wherein the previous/next DNN layer identifiers identify connected DNN layers.

### Claim 6 (depends on 2)

The DNN system of claim 2, wherein the queue packets include at least instructions on how to launch threads, provide a size of private memory allocation, provide a size of group memory allocation, provide a handle for an object in memory that includes an executable ISA image for a computation kernel, and control and synchronization information.

### Claim 7 (depends on 1)

The DNN system of claim 1, wherein certain of the plurality of queues and associated processing elements receive queue packets through remote direct memory access.

### Claim 8 (depends on 1)

The DNN system of claim 1, wherein the plurality of DNN layers are different DNN layer types.

### Claim 9 (depends on 1)

The DNN system of claim 1, wherein each of the multiple inputs is processed at a different DNN layer type.

### Claim 10 (depends on 1)

The DNN system of claim 1, wherein an associated processing element for a queue processes with respect to a specific DNN layer.

### Claim 11 (depends on 10)

The DNN system of claim 10, wherein the specific DNN layer is supported by different DNN networks to enable multiple use of the specific DNN layer.

### Claim 12

A method for deep neural network (DNN) processing, the method comprising:
processing in parallel for multiple inputs:
writing a queue packet associated with each input to a queue, wherein each queue is mapped to one of a plurality of DNN layers in an inference pipeline; and
processing, by a processing element associated with each queue, the input based on a DNN processing profile determined from the queue packet.

### Claim 13 (depends on 12)

The method of claim 12, wherein the queue packet identifies at least a DNN network identifier, a DNN layer identifier, a pointer to buffer for data, and previous/next DNN layer identifiers.

### Claim 14 (depends on 13)

The method of claim 13, wherein the DNN layer identifier identifies a DNN layer type, which is used to determine a nature of computation to be performed and what kernels to launch.

### Claim 15 (depends on 13)

The method of claim 13, wherein the DNN network identifier enables processing of multiple DNN workloads by designating which network to use.

### Claim 16 (depends on 13)

The method of claim 13, wherein the previous/next DNN layer identifiers identify connected DNN layers.

### Claim 17 (depends on 13)

The method of claim 13, wherein the queue packets include at least instructions on how to launch threads, provide a size of private memory allocation, provide a size of group memory allocation, provide a handle for an object in memory that includes an executable ISA image for a computation kernel, and control and synchronization information.

### Claim 18 (depends on 12)

The method of claim 12, further comprising:
writing another queue packet to another queue based on the processed queue packet.

### Claim 19 (depends on 12)

The method of claim 12, wherein the plurality of DNN layers are different DNN layer types.

### Claim 20 (depends on 12)

The method of claim 12, wherein each of the multiple inputs is processed at a different DNN layer type.

### Claim 21 (depends on 12)

The method of claim 12, wherein an associated processing element for a queue processes with respect to a specific DNN layer.

### Claim 22 (depends on 21)

The method of claim 21, wherein the specific DNN layer is supported by different DNN networks to enable multiple use of the specific DNN layer.
