# US 10,372,787 B2 · Hardware accelerator pre-configured with coefficients for matrix-transform operations > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,372,787 B2 - Document ID: US10372787B2 - Status: Granted - Kind: B2 - Application number: 15/839,229 - Filed: 2017-12-12 - Published: 2019-06-13 - Granted: 2019-08-06 - Earliest claimed: 2017-12-12 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Jong Soo Park; Nadav Rotem; Mikhail Smelyanskiy; Abdulkadir Utku Diril - Web page: https://patentgrep.com/patent/US10372787B2 ## CPC - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06F 7/523: Multiplying only - G06F 12/0802: Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches - G06F 12/0875: with dedicated cache, e.g. instruction or stack - G06F 17/156: {using a domain transform, e.g. Fourier transform, polynomial transform, number theoretic transform} - G06N 3/02: Neural networks - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/063: using electronic means - G06N 20/00: Machine learning - Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE ## Abstract A special-purpose hardware accelerator may include a cache configured to store an input matrix related to performing a convolution operation and a matrix-multiplication subsystem pre-configured with matrix-transform coefficients for performing matrix-transform operations. The matrix-multiplication subsystem may perform the convolution operation by (1) reading the input matrix from the cache, (2) transforming the input matrix via matrix multiplication, (3) transforming, via matrix multiplication, a parameter matrix that includes convolution parameters for performing the convolution operation, (4) applying the transformed parameter matrix to the transformed input matrix via an element-wise multiplication operation, and then (5) performing an inverse-transformation operation on the results of the element-wise multiplication operation to create an output matrix for the convolution operation. Various other systems and methods are also disclosed. ## Claims ### Claim 1 A special-purpose hardware accelerator comprising: a cache configured to store an input matrix related to performing a convolution operation; and a matrix-multiplication subsystem pre-configured with matrix-transform coefficients for performing matrix-transform operations, wherein the matrix-multiplication subsystem is configured to perform the convolution operation by: reading the input matrix from the cache; transforming the input matrix via matrix multiplication; transforming, via matrix multiplication, a parameter matrix that comprises convolution parameters for performing the convolution operation; applying the transformed parameter matrix to the transformed input matrix via an element-wise multiplication operation; and performing an inverse-transformation operation on the results of the element-wise multiplication operation to create an output matrix for the convolution operation. ### Claim 2 (depends on 1) The special-purpose hardware accelerator of claim 1, wherein the matrix-multiplication subsystem uses the pre-configured matrix-transform coefficients to at least one of: transform the input matrix; transform the parameter matrix; or perform the inverse-transformation operation on the results of the element-wise multiplication operation. ### Claim 3 (depends on 1) The special-purpose hardware accelerator of claim 1, wherein the pre-configured matrix-transform coefficients comprise at least one of: pre-configured coefficients for transforming the input matrix; pre-configured coefficients for transforming the parameter matrix; or pre-configured coefficients for performing the inverse-transformation operation on the results of the element-wise multiplication operation. ### Claim 4 (depends on 3) The special-purpose hardware accelerator of claim 3, wherein the pre-configured matrix-transform coefficients comprise at least one of: a transposed version of the pre-configured coefficients for transforming the input matrix; a transposed version of the pre-configured coefficients for transforming the parameter matrix; or a transposed version of the pre-configured coefficients for performing the inverse-transformation operation on the results of the element-wise multiplication operation. ### Claim 5 (depends on 1) The special-purpose hardware accelerator of claim 1, wherein the matrix-multiplication subsystem comprises at least one of: a dot-product engine configured to perform the matrix-transform operations; or an element-wise multiplier configured to perform the element-wise multiplication operation. ### Claim 6 (depends on 5) The special-purpose hardware accelerator of claim 5, wherein the matrix-multiplication subsystem transforms the input matrix on-the-fly when reading the input matrix from the cache to the element-wise multiplier. ### Claim 7 (depends on 5) The special-purpose hardware accelerator of claim 5, wherein the parameter matrix is stored in the cache and the matrix-multiplication subsystem transforms the parameter matrix on-the-fly when reading the parameter matrix from the cache to the element-wise multiplier. ### Claim 8 (depends on 1) The special-purpose hardware accelerator of claim 1, wherein the matrix-multiplication subsystem performs the inverse-transformation operation on-the-fly when storing the output matrix to the cache. ### Claim 9 (depends on 1) The special-purpose hardware accelerator of claim 1, wherein: the input matrix comprises the entirety of an input volume for the convolution operation; transforming the input matrix via matrix multiplication comprises transforming the entire input volume via matrix multiplication; and the output matrix comprises the entirety of an output volume for the convolution operation. ### Claim 10 (depends on 1) The special-purpose hardware accelerator of claim 1, wherein: the input matrix comprises an initial portion of an input volume for the convolution operation; transforming the input matrix via matrix multiplication comprises transforming the initial portion of the input volume via matrix multiplication; and creating the output matrix comprises creating an initial portion of an output volume for the convolution operation. ### Claim 11 (depends on 10) The special-purpose hardware accelerator of claim 10, wherein performing the convolution operation further comprises: receiving at least one additional portion of the input volume from the cache; transforming the additional portion of the input volume via matrix multiplication; applying, via an additional element-wise multiplication operation, the transformed parameter matrix to the additional portion of the input volume that was transformed; and performing an additional inverse-transformation operation on the results of the additional element-wise multiplication operation to create an additional portion of the output volume for the convolution operation. ### Claim 12 (depends on 1) The special-purpose hardware accelerator of claim 1, wherein the special-purpose hardware accelerator is configured to: pose each of a plurality of element-wise multiplication operations as a plurality of dot-product operations; and batch the plurality of dot-product operations into a single matrix-multiplication operation for processing by the matrix-multiplication subsystem. ### Claim 13 A computing system comprising: a memory device configured to store an input matrix related to performing a convolution operation; and a special-purpose hardware accelerator comprising a matrix-multiplication subsystem that is pre-configured with matrix-transform coefficients for performing matrix-transform operations, wherein the matrix-multiplication subsystem is configured to perform the convolution operation by: reading the input matrix from the memory device; transforming the input matrix via matrix multiplication; transforming, via matrix multiplication, a parameter matrix that comprises convolution parameters for performing the convolution operation; applying the transformed parameter matrix to the transformed input matrix via an element-wise multiplication operation; and performing an inverse-transformation operation on the results of the element-wise multiplication operation to create an output matrix for the convolution operation. ### Claim 14 (depends on 13) The computing system of claim 13, wherein: the matrix-multiplication subsystem comprises a cache; and reading the input matrix from the memory device comprises: storing the input matrix from the memory device into the cache; and reading the input matrix from the cache. ### Claim 15 (depends on 14) The computing system of claim 14, wherein the matrix-multiplication subsystem comprises at least one of: a dot-product engine configured to perform the matrix-transform operations; or an element-wise multiplier configured to perform the element-wise multiplication operation. ### Claim 16 (depends on 15) The computing system of claim 15, wherein: the matrix-multiplication subsystem transforms the input matrix on-the-fly when reading the input matrix from the cache to the element-wise multiplier; and the parameter matrix is stored in the cache and the matrix-multiplication subsystem transforms the parameter matrix on-the-fly when reading the parameter matrix from the cache to the element-wise multiplier. ### Claim 17 (depends on 14) The computing system of claim 14, wherein the matrix-multiplication subsystem performs the inverse-transformation operation on-the-fly when storing the output matrix to the cache. ### Claim 18 (depends on 13) The computing system of claim 13, wherein the matrix-multiplication subsystem uses the pre-configured matrix-transform coefficients to at least one of: transform the input matrix; transform the parameter matrix; or perform the inverse-transformation operation on the results of the element-wise multiplication operation. ### Claim 19 (depends on 13) The computing system of claim 13, wherein the pre-configured matrix-transform coefficients comprise at least one of: pre-configured coefficients for transforming the input matrix; pre-configured coefficients for transforming the parameter matrix; or pre-configured coefficients for performing the inverse-transformation operation on the results of the element-wise multiplication operation. ### Claim 20 A computer-implemented method comprising: reading, from a cache of a special-purpose hardware accelerator, an input matrix related to performing a convolution operation, wherein the special-purpose hardware accelerator is pre-configured with matrix-transform coefficients for performing matrix-transform operations; and performing, using the special-purpose hardware accelerator, the convolution operation by: transforming the input matrix via matrix multiplication; transforming, via matrix multiplication, a parameter matrix that comprises convolution parameters for performing the convolution operation; applying the transformed parameter matrix to the transformed input matrix via an element-wise multiplication operation; and performing an inverse-transformation operation on the results of the element-wise multiplication operation to create an output matrix for the convolution operation.