# US 2021/0240521 A1 · Architecture to support color scheme-based synchronization for machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0240521 A1
- Document ID: US20210240521A1
- Status: Published application
- Kind: A1
- Application number: 17/237,752
- Filed: 2021-04-22
- Published: 2021-08-05
- Earliest claimed: 2018-05-22
- Assignees: Marvell Asia Pte Ltd
- Applicants: Marvell Asia Pte, Ltd.
- Inventors: Avinash SODANI; Senad DURAKOVIC; Gopal NALAMALAPU
- Web page: https://patentgrep.com/patent/US20210240521A1

## CPC

- G06F 9/30174: {for non-native instruction set, e.g. Javabyte, legacy code}
- G06F 9/4818: {Priority circuits therefor}
- G06F 3/0604: {Improving or facilitating administration, e.g. storage management}
- G06F 3/0659: {Command handling arrangements, e.g. command buffers, queues, command scheduling}
- G06F 3/0673: {Single storage device}
- G06F 9/3879: {for non-native instruction execution, e.g. executing a command; for Java instruction set}
- G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues}
- G06F 9/52: Program synchronisation; Mutual exclusion, e.g. by means of semaphores
- G06F 15/17381: {Two dimensional, e.g. mesh, torus}
- G06N 20/00: Machine learning
- H04N 19/93: Run-length coding
- G06F 9/30087: {Synchronisation or serialisation instructions}
- G06F 9/3869: {Implementation aspects, e.g. pipeline latches; pipeline synchronisation and clocking}
- G06F 9/3871: {Asynchronous instruction pipeline, e.g. using handshake signals between stages}
- G06F 9/522: {Barrier synchronisation}
- G06F 15/7807: {System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package}
- G06F 15/7846: {On-chip cache and off-chip main memory}
- G06F 15/8053: {Vector processors}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 5/04: Inference or reasoning models
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A system to support a machine learning (ML) operation comprises an array-based inference engine comprising a plurality of processing tiles each comprising at least one or more of an on-chip memory (OCM) configured to maintain data for local access by components in the processing tile and one or more processing units configured to perform one or more computation tasks on the data in the OCM by executing a set of task instructions. The system also comprises a data streaming engine configured to stream data between a memory and the OCMs and an instruction streaming engine configured to distribute said set of task instructions to the corresponding processing tiles to control their operations and to synchronize said set of task instructions to be executed by each processing tile, respectively, to wait current certain task at each processing tile to finish before starting a new one.

## Claims

### Claim 1

A system, comprising:
an array-based inference engine comprising a plurality of processing tiles, wherein each processing tile comprises at least one or more of
an on-chip memory (OCM) configured to load and maintain data for local access by components in the processing tile;
one or more processing units configured to perform one or more computation tasks of an operation on data in the OCM by executing a set of task instructions; and
an instruction engine configured to
distribute said set of task instructions to corresponding processing tiles of the inference engine to control their operations via a tile mask having an indicator associated therewith, wherein the tile mask indicates which processing tiles the instructions should be delivered to;
synchronize said set of task instructions to be executed by each processing tile of the plurality of processing tiles, respectively, to wait current task at each processing tile to finish before changing the indicator to a different indicator and distributing a new set of task instructions based on the different indicator.

### Claim 2 (depends on 1)

The system of claim 1, wherein:
each processing unit of the processing units in each processing tile includes one or more of
a first processing unit (POD) configured to perform a dense and/or regular computation task of the operation on the data in the OCM; and
a second processing unit/element (PE) configured to perform a sparse and/or irregular computation task of the operation on the data in the OCM and/or from the POD.

### Claim 3 (depends on 1)

The system of claim 1, wherein:
the instruction engine is configured to pick one processing tile of the plurality of processing tiles that are involved in a synchronization to be a designated tile, which is to inform to the instruction engine when the synchronization is done.

### Claim 4 (depends on 1)

The system of claim 1, wherein:
the instruction engine is configured to utilize a set of synchronization instructions to synchronize the set of task instructions to be executed by the processing units of each processing tile.

### Claim 5 (depends on 4)

The system of claim 4, wherein:
the instruction engine configured to synchronize a prior instruction with a memory access instruction via a sync instruction inserted between the prior instruction and the memory access instruction, wherein the sync instruction synchronizes transferring of the data from the memory to the OCMs in the processing tiles with the tasks being performed in the processing tiles.

### Claim 6 (depends on 1)

The system of claim 1, wherein:
the instruction engine is configured to add an indication to each task instruction to indicate whether the task performed by the task instruction needs to be synchronized.

### Claim 7 (depends on 6)

The system of claim 6, wherein:
the indication is one of: the task needs no synchronization, the task depends locally on a prior task by the processing unit, the task depends on prior task to complete globally among different processing tiles, and the task instruction needs to be held back in the instruction streaming engine until all prior tasks in the inference engine are done.

### Claim 8 (depends on 6)

The system of claim 6, wherein:
the instruction engine is configured to assign every task with a color to represent its current state/status based on its indication.

### Claim 9 (depends on 8)

The system of claim 8, wherein:
the instruction engine is configured to maintain a color transition queue to track color transitions of the tasks, where the color transition queue stores the indications for the color transitions and/or the color of each processing tile after each color transition.

### Claim 10 (depends on 8)

The system of claim 8, wherein:
the inference engine is configured to synchronize data communication among the OCMs when data needs to be transmitted from one processing tile to another based on a color scheme, wherein every processing tile is marked with a color indicating the color of the task the processing tile is ready to execute or is currently executing.

### Claim 11 (depends on 10)

The system of claim 10, wherein:
each processing tile is configured to utilize a color bus that carries color information of its color to other processing tiles connected to it either directly or indirectly.

### Claim 12 (depends on 11)

The system of claim 11, wherein:
the processing tiles marked as a first color have to wait until the processing tiles marked as a second color to finish their tasks before the processing tiles marked as the first color can proceed with their tasks.

### Claim 13

A method, comprising:
distributing a set of task instructions to corresponding processing tiles of an inference engine to control their operations via a tile mask having an indicator to identify which processing tiles the instructions should be delivered to;
performing one or more computation tasks by executing the set of task instructions via the plurality of processing tiles; and
synchronizing the set of task instructions to be executed by each processing tile of the plurality of processing tiles, respectively, to wait current task at each processing tile to finish before starting a new one.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
picking one processing tile of the plurality of processing tiles that are involved in a synchronization to be a designated tile, which is to inform to an instruction engine when the synchronization is done.

### Claim 15 (depends on 13)

The method of claim 13, further comprising:
utilizing a set of synchronization instructions to synchronize the set of task instructions to be executed by processing units of each processing tile.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
synchronizing a prior instruction with a memory access instruction via a sync instruction inserted between the prior instruction and the memory access instruction, wherein the sync instruction synchronizes transferring of the data from a memory to an on-chip memory (OCM) in the processing tiles with the tasks being performed in the processing tiles.

### Claim 17 (depends on 15)

The method of claim 15, further comprising:
adding an indication to each task instruction to indicate whether the task performed by the task instruction needs to be synchronized, wherein the indication is one of: the task needs no synchronization, the task depends locally on a prior task by the processing unit, the task depends on prior task to complete globally among different processing tiles, and the task instruction needs to be held back in the instruction streaming engine until all prior tasks in the inference engine are done.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
assigning every task with a color to represent its current state/status based on its indication.

### Claim 19 (depends on 18)

The method of claim 18, further comprising:
maintaining a color transition queue to track color transitions of the tasks, where the color transition queue stores the indications for the color transitions and/or the color of each processing tile after each color transition.

### Claim 20 (depends on 18)

The method of claim 18, further comprising:
synchronizing data communication among the OCMs when data needs to be transmitted from one processing tile to one another based on a color scheme, wherein every processing tile is marked with a color indicating the color of the task the processing tile is ready to execute or is currently executing.

### Claim 21 (depends on 20)

The method of claim 20, further comprising:
utilizing a color bus that carries color information of its color to other processing tiles connected to it either directly or indirectly.

### Claim 22 (depends on 21)

The method of claim 21, wherein:
the processing tiles marked as a first color have to wait until the processing tiles marked as a second color to finish their tasks before the processing tiles marked as the first color can proceed with their tasks.

### Claim 23

A system, comprising:
a means for distributing a set of task instructions to corresponding processing tiles of an inference engine to control their operations via a tile mask having an indicator to identify which processing tiles the instructions should be delivered to;
a means for performing one or more computation tasks by executing the set of task instructions via the plurality of processing tiles; and
a means for synchronizing the set of task instructions to be executed by each processing tile of the plurality of processing tiles, respectively, to wait current task at each processing tile to finish before starting a new one.
