Back to results
US 12,694,262 B2Granted
Almost-indirect convolution in quantized neural networks
AssigneeSmart Engines Service, LLC
InventorsAnton Vsevolodovich Trusov, Elena Evgenyevna Limonova, Sergey Aleksandrovich Usilin
Life of this patent
Estimating the term…
Abstract
Almost-indirect convolution in quantized neural networks (QNNs). In an embodiment, an indirection buffer is allocated and initialized with pointers to block of values in an input tensor representing an input image. In one or more layers of a QNN, matrix multiplication of a left matrix by a right matrix is performed over a plurality of iterations. In each iteration, a block of one or more rows of the input tensor is packed into a first cache using the indirection buffer, and a convolution kernel is applied to the packed block to produce an output. The outputs are accumulated as an output tensor representing an output image.
Claims11
A method comprising using at least one hardware processor to:
- receive an input tensor having an input height, an input width, and one or more input channels, wherein the input tensor represents an input image;
- allocate and initialize an indirection buffer comprising pointers to blocks of values in the input tensor;
- and,
- in one or more layers of a quantized neural network (QNN),
- perform matrix multiplication of a left matrix by a right matrix over a plurality of iterations, wherein either the left matrix represents the input image and the right matrix represents a convolution kernel or the left matrix represents a transposed convolution kernel and the right matrix represents a transposed image, and wherein the matrix multiplication comprises, in each of the plurality of iterations,
- packing a block of one or more rows of the input tensor into a first cache using the indirection buffer, wherein a number of the one or more rows is less than a number of all rows in the input image, wherein the first cache is an L1 cache of a central processing unit (CPU) of the at least one hardware processor,
- using the at least one hardware processor to store the convolution kernel in an L2 cache of the CPU before performing the matrix multiplication, and
- applying the convolution kernel to the packed block to produce an output, wherein applying the convolution kernel comprises applying a micro-kernel, and wherein the following conditions are satisfied:
- C
-
- K
- h
-
- K
- w
-
- M
- <
- L
- 1
-
- C
-
- K
- h
-
- K
- w
-
- C
- o
- <
- L
- 2
- wherein C is a number of the one or more input channels, Kh is a height of the convolution kernel, Kw is a width of the convolution kernel, M is a height of the micro-kernel, C, is a number of the one or more output channels, L1 is a size of the L1 cache, and L2 is a size of the L2 cache, and
- accumulating the outputs as an output tensor having an output height, an output width, and one or more output channels, wherein the output tensor represents an output image.
33 more elements
A system comprising:
- at least one hardware processor;
- and
- one or more software modules that are configured to, when executed by the at least one hardware processor,
- receive an input tensor having an input height, an input width, and one or more input channels, wherein the input tensor represents an input image, allocate and initialize an indirection buffer comprising pointers to blocks of values in the input tensor, and,
- in one or more layers of a quantized neural network (QNN),
- perform matrix multiplication of a left matrix, representing one of the input image or a convolution kernel, by a right matrix, representing the other one of the input image or the convolution kernel, over a plurality of iterations by, in each of the plurality of iterations,
- packing a block of one or more rows of the input tensor into a first cache using the indirection buffer, wherein a number of the one or more rows is less than a number of all rows in the input image, wherein the first cache is an L1 cache of a central processing unit (CPU) of the at least one hardware processor,
- using the at least one hardware processor to store the convolution kernel in an L2 cache of the CPU before performing the matrix multiplication, and
- applying the convolution kernel to the packed block to produce an output, wherein applying the convolution kernel comprises applying a micro-kernel, and wherein the following conditions are satisfied:
- C
-
- K
- h
-
- K
- w
-
- M
- <
- L
- 1
-
- C
-
- K
- h
-
- K
- w
-
- C
- o
- <
- L
- 2
- wherein C is a number of the one or more input channels, Kh is a height of the convolution kernel, Kw is a width of the convolution kernel, M is a height of the micro-kernel, Co is a number of the one or more output channels, L1 is a size of the L1 cache, and L2 is a size of the L2 cache, and
- accumulating the outputs as an output tensor having an output height, an output width, and one or more output channels, wherein the output tensor represents an output image.
34 more elements
A non-transitory computer-readable medium having instructed stored thereon, wherein the instructions, when executed by a processor, cause the processor to:
- receive an input tensor having an input height, an input width, and one or more input channels, wherein the input tensor represents an input image;
- allocate and initialize an indirection buffer comprising pointers to blocks of values in the input tensor;
- and,
- in one or more layers of a quantized neural network (QNN),
- perform matrix multiplication of a left matrix, representing one of the input image or a convolution kernel, by a right matrix, representing the other one of the input image or the convolution kernel, over a plurality of iterations by, in each of the plurality of iterations,
- packing a block of one or more rows of the input tensor into a first cache using the indirection buffer, wherein a number of the one or more rows is less than a number of all rows in the input image, wherein the first cache is an L1 cache of a central processing unit (CPU) of the at least one hardware processor,
- using the at least one hardware processor to store the convolution kernel in an L2 cache of the CPU before performing the matrix multiplication, and
- applying the convolution kernel to the packed block to produce an output, wherein applying the convolution kernel comprises applying a micro-kernel, and wherein the following conditions are satisfied:
- C
-
- K
- h
-
- K
- w
-
- M
- <
- L
- 1
-
- C
-
- K
- h
-
- K
- w
-
- C
- o
- <
- L
- 2
- wherein C is a number of the one or more input channels, Kh is a height of the convolution kernel, Kw is a width of the convolution kernel, M is a height of the micro-kernel, Co is a number of the one or more output channels, L1 is a size of the L1 cache, and L2 is a size of the L2 cache, and
- accumulating the outputs as an output tensor having an output height, an output width, and one or more output channels, wherein the output tensor represents an output image.
33 more elements
Dependencies are inferred from the claim text.