# US 2019/0244117 A1 · Streaming engine for machine learning architecture

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0244117 A1
- Document ID: US20190244117A1
- Status: Published application
- Kind: A1
- Application number: 16/226,534
- Filed: 2018-12-19
- Published: 2019-08-08
- Earliest claimed: 2018-02-08
- Assignees: Cavium, Inc.
- Applicants: Cavium, LLC
- Inventors: Avinash SODANI; Ulf HANEBUTTE; Senad DURAKOVIC; Hamid Reza GHASEMI; Chia-Hsin CHEN
- Web page: https://patentgrep.com/patent/US20190244117A1

## CPC

- G06F 9/3879: {for non-native instruction execution, e.g. executing a command; for Java instruction set}
- G06F 9/30174: {for non-native instruction set, e.g. Javabyte, legacy code}
- G06F 9/3836: {Instruction issuing, e.g. dynamic instruction scheduling or out of order instruction execution}
- G06F 9/3851: {from multiple instruction streams, e.g. multistreaming}
- G06F 9/3877: {using a secondary processor, e.g. coprocessor (peripheral processor G06F13/12)}
- G06F 15/7807: {System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06F 9/3001: {Arithmetic instructions}
- G06F 15/7864: {on more than one IC chip}
- G06F 15/8023: {Two dimensional arrays, e.g. mesh, torus}
- G06F 2212/602: Details relating to cache prefetching
- G06N 5/04: Inference or reasoning models
- G06N 20/20: Ensemble learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A programmable hardware system for machine learning (ML) includes a core and a streaming engine. The core receives a plurality of commands and a plurality of data from a host to be analyzed and inferred via machine learning. The core transmits a first subset of commands of the plurality of commands that is performance-critical operations and associated data thereof of the plurality of data for efficient processing thereof. The first subset of commands and the associated data are passed through via a function call. The streaming engine is coupled to the core and receives the first subset of commands and the associated data from the core. The streaming engine streams a second subset of commands of the first subset of commands and its associated data to an inference engine by executing a single instruction.

## Claims

### Claim 1

A programmable hardware system for machine learning (ML), comprising:
a core configured to
receive a plurality of commands and data from a host to be analyzed and inferred via machine learning;
transmit each command of a first subset of commands of the plurality of commands for performance-critical operations and associated data thereof to an inference engine for processing via a function call, wherein the each command of the first subset of commands and/or the associated data are encapsulated as parameters in the function call;
an instruction streaming engine coupled to the core and further coupled to the inference engine, wherein the streaming engine is configured to
retrieve and maintain the each command of the first subset of commands and/or the associated data from the function call at a specific location in a buffer;
stream the each command of the first subset of commands and/or its associated data to the inference engine from the buffer; and
said inference engine configured to
retrieve the each command of the first subset of commands and/or its associated data streamed from the buffer; and
perform the performance-critical operations according to the each command of the first subset of commands to analyze and infer a subject from the data.

### Claim 2 (depends on 1)

The programmable hardware system of claim 1, wherein the streaming engine is further configured to receive a streamed inferred data from the inference engine.

### Claim 3 (depends on 2)

The programmable hardware system of claim 2, wherein the streaming engine is further configured to stream the received inferred data to the core.

### Claim 4 (depends on 1)

The programmable hardware system of claim 1, wherein the streaming engine comprises:
an instruction streaming engine coupled to the core, wherein the instruction streaming engine is configured to stream the first subset of commands to the inference engine; and
a data steaming engine coupled to the inference engine and configured to generate one or more streams of data associated with the first subset of commands, and wherein the data streaming engine is configured to stream the one or more streams of data to the inference engine be analyzed and inferred.

### Claim 5 (depends on 1)

The programmable hardware system of claim 1, wherein the streaming engine is configured to stream instructions to the inference engine in an instruction set architecture that is different from an instruction set architecture format received from the core.

### Claim 6 (depends on 1)

The programmable hardware system of claim 1, wherein the buffer is coupled to the core and further coupled to the streaming engine, wherein the core continuously writes to the buffer until a certain condition is met, and wherein the streaming engine continuously reads from the buffer until another certain condition is met.

### Claim 7 (depends on 6)

The programmable hardware system of claim 6, wherein the certain condition is when available buffer associated with the buffer is below a threshold value.

### Claim 8 (depends on 7)

The programmable hardware system of claim 7, wherein the available buffer is tracked using a head pointer maintained by the core locally, and wherein the head pointer is incremented each time the core writes to the buffer and the available buffer associated with the buffer is decremented each time the core writes to the buffer.

### Claim 9 (depends on 7)

The programmable hardware system of claim 7, wherein the core reads a value stored in a memory mapped input/output (MIMO) responsive to the certain condition being met, wherein the MIMO stores a value of the head pointer and a tail pointer associated with a location the streaming engine reads from the buffer, and wherein the core is configured to set the available buffer size.

### Claim 10 (depends on 9)

The programmable hardware system of claim 9, wherein the core is configured to set the available buffer size to the tail pointer minus the head pointer and result thereof modulo actual size of the buffer.

### Claim 11 (depends on 6)

The programmable hardware system of claim 6, wherein the another certain condition is when buffer size to read from is greater than zero.

### Claim 12 (depends on 11)

The programmable hardware system of claim 11, wherein the buffer size to read from is tracked using a tail pointer maintained by the streaming engine locally, and wherein the tail pointer is incremented each time the streaming engine reads from the buffer and wherein the buffer size to read from is the tail pointer minus a head pointer and result thereof modulo actual size of the buffer, wherein the head pointer is maintained by the core locally and incremented each time the core writes to the buffer.

### Claim 13 (depends on 1)

The programmable hardware system of claim 1 wherein the core maintains a head pointer where the core writes to and wherein the streaming engine maintains a tail pointer where the streaming engine reads from, and wherein the head pointer and the tail pointer are stored in a memory mapped input/output (MMIO) space that is mapped into registers in the streaming engine.

### Claim 14 (depends on 1)

The programmable hardware system of claim 1, wherein the buffer is a circular buffer allocated in a DDR memory, and wherein a size of the buffer is fixed a-priori at compile time.

### Claim 15

A programmable hardware system for machine learning (ML), comprising:
a core configured to receive a plurality of commands and a plurality of data from a host to be analyzed and inferred via machine learning, wherein the core is further configured to transmit a first subset of commands of the plurality of commands that is performance-critical operations and associated data thereof of the plurality of data for efficient processing thereof, wherein the first subset of commands and the associated data are passed through via a function call; and
a streaming engine coupled to the core configured to receive the first subset of commands and the associated data from the core, and wherein the streaming engine is configured to stream a second subset of commands of the first subset of commands and its associated data to an inference engine by executing a single instruction.

### Claim 16 (depends on 15)

The programmable hardware system of claim 15, wherein the streaming engine is further configured to receive a streamed inferred data from the inference engine and further configured to stream the received inferred data to the core.

### Claim 17 (depends on 15)

The programmable hardware system of claim 15, wherein the streaming engine comprises:
an instruction streaming engine coupled to the core, wherein the instruction streaming engine is configured to stream the first subset of commands to the inference engine; and
a data steaming engine coupled to the inference engine and configured to generate one or more streams of data associated with the first subset of commands, and wherein the data streaming engine is configured to stream the one or more streams of data to the inference engine be analyzed and inferred.

### Claim 18 (depends on 15)

The programmable hardware system of claim 15, wherein the streaming engine is configured to stream instructions to the inference engine in an instruction set architecture that is different from an instruction set architecture format received from the core.

### Claim 19 (depends on 15)

The programmable hardware system of claim 15 further comprising:
a buffer coupled to the core and further coupled to the streaming engine, wherein the core continuously writes to the buffer until a certain condition is met, and wherein the streaming engine continuously reads from the buffer until another certain condition is met.

### Claim 20

A method comprising:
receive a plurality of commands and data to be analyzed and inferred via machine learning;
transmitting each command of a first subset of commands of the plurality of commands for performance-critical operations and associated data thereof to an inference engine for processing via a function call, wherein the each command of the first subset of commands and/or the associated data are encapsulated as parameters in the function call;
retrieving and maintaining the each command of the first subset of commands and/or the associated data from the function call at a specific location in a buffer;
streaming the each command of the first subset of commands and/or its associated data to the inference engine from the buffer; and
retrieving the each command of the first subset of commands and/or its associated data streamed from the buffer; and
performing the performance-critical operations according to the each command of the first subset of commands to analyze and infer a subject from the data.

### Claim 21 (depends on 20)

The method of claim 20 further comprising receiving a streamed inferred data from the inference engine.

### Claim 22 (depends on 21)

The method of claim 21 further comprising streaming the received inferred data to a core.

### Claim 23 (depends on 20)

The method of claim 20 further comprising:
continuously writing to the buffer until a certain condition is met; and
continuously reading from the buffer until another certain condition is met.

### Claim 24 (depends on 23)

The method of claim 23, wherein the certain condition is when available buffer associated with the buffer is below a threshold value.

### Claim 25 (depends on 24)

The method of claim 24 further comprising:
tracking the available buffer using a head pointer maintained by a core locally;
incrementing the head pointer each time the core writes to the buffer; and
decrementing the available buffer each time the core writes to the buffer.

### Claim 26 (depends on 25)

The method of claim 25 further comprising:
reading a value stored in a memory mapped input/output (MIMO) responsive to the certain condition being met, wherein the MIMO stores a value of a head pointer maintained by a core locally where the core is writing to in the buffer and further the MIMO stores a tail pointer associated with a location a streaming engine reads from the buffer.

### Claim 27 (depends on 26)

The method of claim 26 further comprising setting the available buffer size to the tail pointer minus the head pointer and result thereof modulo actual size of the buffer.

### Claim 28 (depends on 23)

The method of claim 23, wherein the another certain condition is when buffer size to read from is greater than zero.

### Claim 29 (depends on 20)

The method of claim 20 further comprising:
tracking buffer size to read from using a tail pointer, wherein the tail pointer is maintained by a streaming engine locally;
incrementing the tail pointer each time a streaming engine reads from the buffer; and
setting the buffer size to read from to the tail pointer minus a head pointer and result thereof modulo actual size of the buffer, wherein the head pointer is maintained by the core locally and incremented each time the buffer is written to.
