# US 2021/0334072 A1 · Mapping convolution to connected processing elements using distributed pipelined separable convolution operations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334072 A1
- Document ID: US20210334072A1
- Status: Published application
- Kind: A1
- Application number: 16/855,927
- Filed: 2020-04-22
- Published: 2021-10-28
- Earliest claimed: 2020-04-22
- Applicants: Facebook, Inc.
- Inventors: Rakesh Komuravelli; Krishnakumar Narayanan Nair; Abdulkadir Utku Diril; Ehsan Khish Ardestani Zadeh; Yuchen Hao; Martin Schatz; Thomas Mark Ulrich; Olivia Wu; Anup Ramesh Kadkol; Amin Firoozshahian
- Web page: https://patentgrep.com/patent/US20210334072A1

## CPC

- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 7/50: Adding; Subtracting (G06F7/483 - G06F7/491, G06F7/544 - G06F7/556 take precedence)
- G06F 7/523: Multiplying only
- G06F 17/153: {Multidimensional correlation or convolution}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06N 20/00: Machine learning
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A processor system comprises a plurality of dot product processor units and element-wise multiplication units. The dot product processor units perform a depthwise convolution of a data matrix with a separate depthwise convolution weight matrix for each data matrix channel. Each dot product processor unit performs at least a portion of the depthwise convolution for one or more data matrix channels. The element-wise multiplication units perform multiplication operations of a pointwise convolution. Each element-wise multiplication unit applies to each depthwise convolution partial result element received from one or more of the dot product processor units a corresponding data element from each of a plurality of pointwise convolution weight filters to determine element-wise multiplication unit results. The processor system sums together different groups of data elements from the element-wise multiplication unit results to at least in part calculate different data elements of a result of the pointwise convolution.

## Claims

### Claim 1

A processor system, comprising:
a plurality of dot product processor units configured to perform a depthwise convolution of a data matrix having a plurality of channels with a plurality of depthwise convolution weight matrices including a separate depthwise convolution weight matrix for each of the plurality of channels, wherein each of the dot product processor units is configured to perform at least a portion of the depthwise convolution for one or more channels included in the plurality of channels; and
a plurality of element-wise multiplication units configured to at least in part perform multiplication operations of a pointwise convolution, wherein each of the element-wise multiplication units is configured to apply to each depthwise convolution partial result element received from one or more of the dot product processor units a corresponding data element from each of a plurality of pointwise convolution weight filters to determine element-wise multiplication unit results;
wherein the processor system is configured to sum together different groups of data elements from the element-wise multiplication unit results from the plurality of element-wise multiplication units to at least in part calculate different data elements of a result of the pointwise convolution.

### Claim 2 (depends on 1)

The system of claim 1, wherein the plurality of element-wise multiplication units is configured to at least in part perform the multiplication operations of the pointwise convolution prior to a completion of the depthwise convolution.

### Claim 3 (depends on 1)

The system of claim 1, wherein the processor system is configured to sum together the different groups of the data elements from the element-wise multiplication unit results at least in part in parallel.

### Claim 4 (depends on 1)

The system of claim 1, wherein each of the dot product processor units includes a plurality of calculation units.

### Claim 5 (depends on 4)

The system of claim 4, wherein each calculation unit of the plurality of calculation units includes a vector multiply unit and a vector adder unit.

### Claim 6 (depends on 5)

The system of claim 5, wherein the vector adder unit includes an adder tree.

### Claim 7 (depends on 1)

The system of claim 1, wherein the data matrix is a three-dimensional machine learning data matrix.

### Claim 8 (depends on 1)

The system of claim 1, wherein the separate depthwise convolution weight matrix and each of the plurality of pointwise convolution weight filters are machine learning weight matrices.

### Claim 9 (depends on 1)

The system of claim 1, wherein the separate depthwise convolution weight matrix is a 3x3 matrix.

### Claim 10 (depends on 1)

The system of claim 1, wherein the separate depthwise convolution weight matrix is a 3×3, 5×5, 7×7, 9×9, or 11×11 matrix.

### Claim 11 (depends on 1)

The system of claim 1, wherein each of the plurality of pointwise convolution weight filters has a channel depth that corresponds to a count of the plurality of channels of the data matrix.

### Claim 12 (depends on 1)

The system of claim 1, further comprising:
a plurality of reduction units;
is a plurality of point-to-point connections, wherein each point-to-point connection of the plurality of point-to-point connections is configured to provide a result of a first reduction unit of the plurality of reduction units to a second reduction unit of the plurality of reduction units; and
a communication bus connecting together the plurality of dot product processor units.

### Claim 13 (depends on 12)

The system of claim 12, wherein the first reduction unit includes an adder configured to perform vector addition operations.

### Claim 14 (depends on 12)

The system of claim 12, wherein each of the plurality of dot product processor units is configured to receive a depthwise convolution operation instruction via the communication bus.

### Claim 15 (depends on 12)

The system of claim 12, wherein each of the plurality of element-wise multiplication units is configured to receive a pointwise convolution operation instruction via the communication bus.

### Claim 16 (depends on 12)

The system of claim 12, wherein the second reduction unit of the plurality of reduction units is configured to add together a local result of an element-wise multiplication unit of the plurality of element-wise multiplication units with a reduced result of the first reduction unit of the plurality of reduction units to determine a reduction unit result.

### Claim 17 (depends on 16)

The system of claim 16, wherein the second reduction unit is further configured to provide the reduction unit result to a third reduction unit of the plurality of reduction units via a point-to-point connection of the plurality of point-to-point connections.

### Claim 18

A method comprising:
determining a vector of depthwise convolution partial result elements using a dot product engine of a first processing element, wherein the vector of depthwise convolution partial result elements corresponds to a matrix slice from an assigned channel of a three-dimensional data matrix and a separate depthwise convolution weight matrix;
providing the vector of depthwise convolution partial result elements to an element-wise multiplication unit of the first processing element;
determining element-wise multiplication results for each element of the vector of depthwise convolution partial result elements by performing multiplication operations of a pointwise convolution using the each element and corresponding data elements from a channel of a plurality of pointwise convolution weight filters;
providing the element-wise multiplication results for each element of the vector of depthwise convolution partial result elements to a reduction unit of the first processing element;
receiving upstream results from a second processing element via a first point-to-point connection;
summing together the upstream results with the corresponding element-wise multiplication results to determine reduction unit results; and
sending the reduction unit results to a third processing element via a second point-to-point connection.

### Claim 19 (depends on 18)

The method of claim 18, wherein the upstream results are at least in part determined using corresponding data elements from corresponding channels of the plurality of pointwise convolution weight filters.

### Claim 20

A processing element system, comprising:
a dot product processor unit configured to perform a depthwise convolution using a two-dimensional matrix slice of a three-dimensional data matrix with a depthwise convolution weight matrix of a plurality of depthwise convolution weight matrices;
an element-wise multiplication unit configured to at least in part perform multiplication operations of a pointwise convolution by applying to each depthwise convolution partial result element received from the dot product processor unit a corresponding data element from each of a plurality of pointwise convolution weight filters to determine local element-wise multiplication unit results;
a first point-to-point connection configured to receive an upstream result from an upstream processing element;
a reduction unit configured to sum together the received upstream result and the determined local element-wise multiplication unit results to determine a reduction unit result; and
a second point-to-point connection configured to provide the determined reduction unit result to a downstream processing element.
