# US 11,443,013 B2 · Pipelined pointwise convolution using per-channel convolution operations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,013 B2 - Document ID: US11443013B2 - Status: Granted - Kind: B2 - Application number: 16/826,697 - Filed: 2020-03-23 - Published: 2021-09-23 - Granted: 2022-09-13 - Earliest claimed: 2020-03-23 - Assignees: Meta Platforms, Inc. - Applicants: Meta Platforms, Inc. - Inventors: Rakesh Komuravelli; Krishnakumar Narayanan Nair; Abdulkadir Utku Diril; Ehsan Khish Ardestani Zadeh; Yuchen Hao; Martin Schatz; Thomas Mark Ulrich; Olivia Wu; Anup Ramesh Kadkol; Amin Firoozshahian - Web page: https://patentgrep.com/patent/US11443013B2 ## CPC - G06F 17/153: {Multidimensional correlation or convolution} - G06N 3/063: using electronic means - G06F 9/3001: {Arithmetic instructions} - G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations} - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A processor system comprises a hardware channel convolution processor unit and dot product processor unit. The channel convolution processor unit is configured to perform depthwise convolution, including by multiplying each data element of a first group of data elements of a convolution data matrix with a corresponding data element of a second group of data elements of a plurality of depthwise convolution weight matrices and summing together, for each specific channel, multiplication results corresponding to the specific channel to determine one corresponding result data element in a corresponding channel convolution result matrix to calculate a portion of depthwise convolution results. The dot product processor unit is configured to perform pointwise convolution, including applying pointwise weight matrices to the portion of depthwise convolution results to determine a portion of separable convolution results while at least another portion of the depthwise convolution results is being calculated by the processor system. ## Claims ### Claim 1 A processor system, comprising: a hardware channel convolution processor unit configured to perform a portion of a depthwise convolution, including by being configured to: multiply each data element of a first group of data elements of a plurality of channels of a portion of a convolution data matrix with a corresponding data element of a second group of data elements of a plurality of depthwise convolution weight matrices to determine a corresponding multiplication result of multiplication results; and for each specific channel of the plurality of channels, sum together ones of the multiplication results corresponding to the specific channel to determine one corresponding channel convolution result data element in a corresponding channel convolution result matrix to calculate a portion of depthwise convolution results; and a hardware dot product processor unit configured to perform at least a portion of a pointwise convolution, including by being configured to apply pointwise weight matrices to the portion of depthwise convolution results calculated using the hardware channel convolution processor unit to determine a portion of separable convolution results while at least another portion of the depthwise convolution results is being calculated in parallel by the hardware channel convolution processor unit of the processor system; wherein the hardware channel convolution processor unit and the hardware dot product processor unit are different hardware units and the portion of depthwise convolution results is provided from the hardware channel convolution processor unit to the hardware dot product processor unit for pipeline processing. ### Claim 2 (depends on 1) The system of claim 1, further comprising a first processing element and a second processing element, and wherein the first processing element includes the hardware channel convolution processor unit and the second processing element includes the hardware dot product processor unit. ### Claim 3 (depends on 1) The system of claim 1, wherein a total count of the data elements of the first group is the same as a total count of the data elements of the second group. ### Claim 4 (depends on 1) The system of claim 1, wherein the hardware channel convolution processor unit comprises a plurality of calculation units and each calculation unit of the plurality of calculation units is configured to receive a plurality of data elements of the first group corresponding to a same channel of the convolution data matrix and a plurality of corresponding data elements of the second group corresponding to a separate convolution weight matrix for the same channel of the convolution data matrix. ### Claim 5 (depends on 4) The system of claim 4, wherein each calculation unit of the plurality of calculation units includes a different vector multiply unit and a different vector adder unit. ### Claim 6 (depends on 5) The system of claim 5, wherein each of the different vector adder units includes a different adder tree. ### Claim 7 (depends on 1) The system of claim 1, wherein the convolution data matrix is a three-dimensional machine learning data matrix. ### Claim 8 (depends on 1) The system of claim 1, further comprising a data input unit configured to: process the data elements of the first group by channel into a plurality of data input vectors, wherein each of the plurality of data input vectors includes data elements corresponding to a two-dimensional sub-matrix of the convolution data matrix. ### Claim 9 (depends on 1) The system of claim 1, further comprising a weight input unit configured to: process the data elements of the second group into a plurality of weight input vectors, wherein each of the plurality of weight input vectors includes data elements corresponding to one of the plurality of depthwise convolution weight matrices. ### Claim 10 (depends on 1) The system of claim 1, wherein each of the plurality of depthwise convolution weight matrices is a 3×3, 5×5, 7×7, 9×9, or 11×11 matrix. ### Claim 11 (depends on 1) The system of claim 1, wherein the data elements of the first group are 4-bit, 8-bit, 2-byte, or 4-byte elements. ### Claim 12 (depends on 1) The system of claim 1, wherein a total count of the data elements of the first group is a multiple of a cache line size. ### Claim 13 (depends on 1) The system of claim 1, wherein a total count of the pointwise weight matrices is a multiple of a cache line size. ### Claim 14 (depends on 1) The system of claim 1, wherein the portion of depthwise convolution results calculated using the hardware channel convolution processor unit is stored in a local memory of a processing element. ### Claim 15 (depends on 1) The system of claim 1, wherein the portion of depthwise convolution results calculated using the hardware channel convolution processor unit to determine the portion of separable convolution results has a width and height dimension of 1×1. ### Claim 16 (depends on 15) The system of claim 15, wherein a depth dimension of the portion of depthwise convolution results calculated using the hardware channel convolution processor unit to determine the portion of separable convolution results is a multiple of a cache line size of a memory. ### Claim 17 (depends on 1) The system of claim 1, wherein the convolution data matrix, the plurality of depthwise convolution weight matrices, the portion of depthwise convolution results, the pointwise weight matrices, and the portion of separable convolution results are arranged in a depth-first layout format. ### Claim 18 A method, comprising: receiving a three-dimensional convolution operation instruction specifying a convolution data matrix; identifying separate convolution operations from the received three-dimensional convolution operation instruction; using a hardware channel convolution processor unit to: perform a depthwise convolution operation with a portion of the convolution data matrix and a plurality of depthwise convolution weight matrices; and determine a portion of depthwise convolution results; and using a hardware dot product processor unit to: perform a pipelined pointwise convolution operation using the portion of depthwise convolution results and a plurality of pointwise weight matrices while at least another portion of the depthwise convolution results is being calculated in parallel by the hardware channel convolution processor unit; and determine separable convolution results; wherein the hardware channel convolution processor unit and the hardware dot product processor unit are different hardware units and the portion of depthwise convolution results is provided from the hardware channel convolution processor unit to the hardware dot product processor unit for pipeline processing. ### Claim 19 (depends on 18) The method of claim 18, further comprising storing the separable convolution results to a memory location using a same layout format as the convolution data matrix. ### Claim 20 A method, comprising: for each data element of a first group of data elements of a plurality of channels of a portion of a convolution data matrix, multiplying at a hardware channel convolution processor unit the data element of the first group of data elements with a corresponding data element of a second group of data elements of a plurality of depthwise convolution weight matrices to determine a corresponding multiplication result of multiplication results; for each specific channel of the plurality of channels, summing together at the hardware channel convolution processor unit ones of the multiplication results corresponding to the specific channel to determine one corresponding channel convolution result data element in a corresponding channel convolution result matrix to calculate a portion of depthwise convolution results; transmitting the portion of depthwise convolution results from the hardware channel convolution processor unit to a hardware dot product processor unit for pipeline processing; and applying at the hardware dot product processor unit pointwise weight matrices to the portion of depthwise convolution results calculated using the hardware channel convolution processor unit to determine a portion of separable convolution results while at least another portion of the depthwise convolution results is being calculated in parallel by the hardware channel convolution processor unit; wherein the hardware channel convolution processor unit and the hardware dot product processor unit are different hardware units U.