# US 12,118,451 B2 · Deep convolutional network heterogeneous architecture

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,451 B2
- Document ID: US12118451B2
- Status: Granted
- Kind: B2
- Application number: 15/423,272
- Filed: 2017-02-02
- Published: 2018-07-05
- Granted: 2024-10-15
- Earliest claimed: 2017-01-04
- Assignees: STMicroelectronics S.r.l.; STMicroelectronics International N.V.
- Applicants: Stmicroelectronics S.R.L.; Stmicroelectronics International N.V.
- Inventors: Giuseppe Desoli; Thomas Boesch; Nitin Chawla; Surinder Pal Singh; Elio Guidetti; Fabio Giuseppe De Ambroggi; Tommaso Majo; Paolo Sergio Zambotti
- Web page: https://patentgrep.com/patent/US12118451B2

## CPC

- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06F 30/327: Logic synthesis; Behaviour synthesis, e.g. mapping logic, HDL to netlist, high-level language to RTL or netlist
- G06F 30/34: for reconfigurable circuits, e.g. field programmable gate arrays [FPGA] or programmable logic devices [PLD]
- G06F 30/347: Physical level, e.g. placement or routing
- G06N 3/045: Combinations of networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 20/00: Machine learning
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06F 9/44505: {Configuring for program initiating, e.g. using registry, configuration files}
- G06F 13/4022: {using switching circuits, e.g. switching matrix, connection or expansion network (G06F13/4009 takes precedence)}
- G06F 15/7817: {Specially adapted for signal processing, e.g. Harvard architectures}
- G06F 2115/02: System on chip [SoC] design
- G06F 2115/08: Intellectual property [IP] blocks or IP cores
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/063: using electronic means
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments are directed towards a system on chip (SoC) that implements a deep convolutional network heterogeneous architecture. The SoC includes a system bus, a plurality of addressable memory arrays coupled to the system bus, at least one applications processor core coupled to the system bus, and a configurable accelerator framework coupled to the system bus. The configurable accelerator framework is an image and deep convolutional neural network (DCNN) co-processing system. The SoC also includes a plurality of digital signal processors (DSPs) coupled to the system bus, wherein the plurality of DSPs coordinate functionality with the configurable accelerator framework to execute the DCNN.

## Claims

### Claim 1

A system on chip (SoC) that implements a deep convolutional neural network architecture, the SoC comprising:
a system bus;
a plurality of addressable memory arrays coupled to the system bus;
at least one applications processor core coupled to the system bus;
a configurable accelerator framework coupled to the system bus, wherein the configurable accelerator framework is an image and deep convolutional neural network (DCNN) co-processing system that includes a plurality of convolution accelerators configured to perform convolutional operations; and
a plurality of digital signal processors (DSPs) coupled to the system bus, wherein the plurality of DSPs are configured to perform non-convolution data processing operations associated with processing of a DCNN kernel in parallel with execution of convolutional operations associated with processing of the DCNN kernel by the plurality of convolution accelerators of the configurable accelerator framework to execute the DCNN kernel, wherein the non-convolutional data processing operations associated with the DCNN kernel include one or more operations selected from a group of operations consisting of pooling, non-linear activation and cross-channel response normalization.

### Claim 2 (depends on 1)

The SoC of claim 1, wherein the plurality of digital signal processors include a plurality of DSP clusters coupled to the system bus, and wherein each DSP cluster of the plurality of DSP clusters includes at least two separate DSPs.

### Claim 3 (depends on 2)

The SoC of claim 2, further comprising:
a global DSP cluster crossbar switch coupled to each of the plurality of DSP clusters and to the system bus.

### Claim 4 (depends on 2)

The SoC of claim 2, wherein each DSP cluster includes:
a DSP cluster crossbar switch;
a first local DSP crossbar switch and a second local DSP crossbar switch, each coupled to the DSP cluster crossbar switch;
a first DSP, a first instruction cache, and a first DSP memory, each coupled to the first local DSP crossbar switch; and
a second DSP, a second instruction cache, and a second DSP memory, each coupled to the second local DSP crossbar switch.

### Claim 5 (depends on 2)

The SoC of claim 2, wherein each DSP cluster includes:
a DSP cluster crossbar switch;
a first local DSP crossbar switch and a second local DSP crossbar switch, each coupled to the DSP cluster crossbar switch;
a first DSP coupled to the first local DSP crossbar switch;
a second DSP coupled to the second local DSP crossbar switch;
a shared DSP cluster memory coupled to the DSP cluster crossbar switch; and
a direct memory access coupled between the shared DSP cluster memory and the system bus.

### Claim 6 (depends on 1)

The SoC of claim 1, wherein the configurable accelerator framework includes:
a reconfigurable stream switch coupled to the system bus; and
wherein the plurality of convolution accelerators are coupled to the stream switch.

### Claim 7 (depends on 6)

The SoC of claim 6, wherein the stream switch is configurable at run-time and reconfigurable during execution of the at least one DCNN.

### Claim 8 (depends on 6)

The SoC of claim 6, wherein each of the plurality of convolution accelerators is configurable at run-time and reconfigurable during execution of the DCNN kernel.

### Claim 9 (depends on 1)

The SoC of claim 1, wherein non-convolution operations associated with the DCNN kernel are synchronized with convolutional operations associated with the DCNN kernel using interrupts.

### Claim 10 (depends on 1)

The SoC of claim 1, wherein non-convolution operations associated with the DCNN kernel are synchronized with convolutional operations associated with the DCNN kernel using mailboxes.

### Claim 11 (depends on 1)

The SoC of claim 1, wherein the memory arrays have a hierarchical structure.

### Claim 12 (depends on 6)

The SoC of claim 6 wherein the configurable accelerator framework includes a plurality of control registers and the stream switch, in operation, provides the plurality of DSPs with access to the plurality of control registers during processing of the DCNN kernel to control performing of non-convolution operations by the plurality of DSPs.

### Claim 13

A mobile computing device comprising:
an imaging sensor that captures images;
a system on chip (SoC) that implements a deep convolutional neural network architecture, the SoC, including:
an SoC bus;
an on-chip memory coupled to the SoC bus;
a configurable accelerator framework coupled to the SoC bus, the configurable accelerator framework including a reconfigurable dataflow accelerator fabric that receives the captured images from the imaging sensor for deep convolutional neural network (DCNN) processing by at least one convolution accelerator; and
a plurality of digital signal processor (DSPs) clusters coupled to the SoC bus, the plurality of DSP clusters configured to perform non-convolution operations for the DCNN and arranged to coordinate functionality with the at least one convolution accelerator of the configurable accelerator framework to execute the DCNN, wherein the plurality of DSP clusters are configured to perform the non-convolutional data processing operations for a DCNN kernel in parallel with execution of convolutional operations for the DCNN kernel by the at least one convolutional accelerator, wherein the non-convolutional data processing operations for the DCNN kernel include one or more operations selected from a group of operations consisting of pooling, non-linear activation and cross-channel response normalization.

### Claim 14 (depends on 13)

The mobile computing device of claim 13, wherein each DSP cluster of the plurality of DSP clusters includes at least two separate DSPs arranged for communication with each other and for communication with the SoC bus via a DSP cluster crossbar switch.

### Claim 15 (depends on 13)

The mobile computing device of claim 13, comprising:
a global DSP cluster crossbar switch coupled to each of the plurality of DSP clusters and to the SoC bus.

### Claim 16 (depends on 13)

The mobile computing device of claim 13, wherein each DSP cluster includes:
a DSP cluster crossbar switch;
a first local DSP crossbar switch and a second local DSP crossbar switch, each coupled to the DSP cluster crossbar switch;
a first DSP, a first instruction cache, and a first DSP memory, each coupled to the first local DSP crossbar switch; and
a second DSP, a second instruction cache, and a second DSP memory, each coupled to the second local DSP crossbar switch.

### Claim 17 (depends on 13)

The mobile computing device of claim 13, wherein each DSP cluster includes:
a DSP cluster crossbar switch;
a first local DSP crossbar switch and a second local DSP crossbar switch, each coupled to the DSP cluster crossbar switch;
a first DSP coupled to the first local DSP crossbar switch;
a second DSP coupled to the second local DSP crossbar switch;
a shared DSP cluster memory coupled to the DSP cluster crossbar switch; and
a direct memory access coupled between the shared DSP cluster memory and the SoC bus.

### Claim 18 (depends on 13)

The mobile computing device of claim 13, wherein the reconfigurable dataflow accelerator fabric is configurable at run-time and reconfigurable during execution of the DCNN.

### Claim 19 (depends on 13)

The mobile computing device of claim 13, wherein the at least one convolution accelerator is configurable at run-time and reconfigurable during execution of the DCNN.

### Claim 20 (depends on 13)

The mobile computing device of claim 13 wherein the configurable accelerator framework includes a plurality of control registers and a stream switch, which, in operation, provides the plurality of DSPs with access to the plurality of control registers during processing of the DCNN kernel to control performing of non-convolution operations by the plurality of DSPs.

### Claim 21

A system on a chip (SoC), comprising:
a communication bus;
a memory coupled to the communication bus;
a configurable accelerator framework coupled to the communication bus, the configurable accelerator framework having a reconfigurable stream switch coupled to the communication bus and a plurality of convolution accelerators coupled to the stream switch, the plurality of convolution accelerators arranged to perform convolution operations on image data during execution of a deep convolutional neural network (DCNN); and
a plurality of digital signal processors (DSPs) coupled to the communication bus, wherein the plurality of DSPs perform non-convolution operations for the DCNN and coordinate functionality with the plurality of convolution accelerators of the configurable accelerator framework to execute the DCNN, wherein the plurality of DSP are configured to perform non-convolutional data processing operations for a DCNN kernel in parallel with execution of convolutional operations for the DCNN kernel by the plurality of convolutional accelerators, wherein the non-convolutional data processing operations for the DCNN kernel include one or more operations selected from a group of operations consisting of pooling, non-linear activation and cross-channel response normalization.

### Claim 22 (depends on 21)

The SoC of claim 21, wherein the plurality of DSPs are in clusters, each cluster having at least two separate DSPs, each cluster including a cluster communication bus coupled to the communication bus.

### Claim 23 (depends on 22)

The SoC of claim 22, comprising:
a global DSP communication bus coupled to the communication bus and coupled to the cluster communication bus of each cluster;
wherein each cluster includes:
a first local DSP communication bus and a second local DSP communication bus, each of the first and second local DSP communication buses coupled to the cluster communication bus;
a first DSP coupled to the first local DSP communication bus;
a second DSP coupled to the second local DSP communication bus;
a shared cluster memory coupled to the cluster communication bus; and
a direct memory access coupled between the shared cluster memory and the communication bus.

### Claim 24 (depends on 21)

The SoC of claim 21, wherein the configurable accelerator framework includes a plurality of control registers to store control information to control execution of the DCNN, and the reconfigurable stream switch, in operation, provides DSPs of the plurality of DSPs and convolutional accelerators of the plurality of convolutional accelerators with access to the plurality of control registers.