Configurable Convolution Engine
AssigneeApple Inc.
InventorsSuk Hwan Lim, Junji Sugisawa, Muge Wang
Life of this patent
Estimating the term…
Abstract
Embodiments of the present disclosure relate to a configurable convolution engine that receives configuration information to perform convolution or its variant operations on streaming input data of various formats. To process streaming input data, input data of multiple channels are received and stored in an input buffer circuit in an interleaved manner. Data values of the interleaved input data are retrieved and forwarded to multiplier circuits where multiplication with a corresponding filter element of a kernel is performed. Varying number of kernels with different sizes and sparsity can also be used for the convolution operations.
Claims26
A convolution engine, comprising:
- an input buffer circuit configured to receive and store data values of a plurality of channels of input data in an interleaved manner, the plurality of channels of input data including at least a first channel of input data and a second channel of input data interleaved with the first channel of input data;
- a datapath switch circuit configured to retrieve, from the input buffer circuit, data values of the first channel of input data and skip data values of the second channel of input data in a cycle, and retrieve the data values of the second channel of input data and skip the data values of the first channel of input data in another cycle;
- a filter switch circuit configured to retrieve filter elements of at least one kernel for performing a convolution operation;
- and
- a computation core circuit configured to:
- receive the data values from the datapath switch and the retrieved filter elements from the filter switch;
- multiply each of the data values with a corresponding filter element to obtain multiplied values;
- and
- process subsets of multiplied values to obtain output values.
A method of performing convolution, comprising:
- storing interleaved data values of a plurality of channels of input data in an input buffer circuit, the plurality of channels of input data including at least a first channel of input data and a second channel of input data interleaved with the first channel of input data;
- retrieving, by a datapath switch circuit and from the input buffer circuit, data values of the first channel of input data and skipping data values of the second channel of input data in a cycle, and retrieving the data values of the second channel of input data and skipping the data values of the first channel of input data in another cycle;
- retrieving, by a filter switch circuit, filter elements of at least one kernel for performing a convolution operation;
- receiving, by a computation core circuit, the data values from the datapath switch and the retrieved filter elements from the filter switch;
- multiplying, by the computation core circuit, each of the data values with a corresponding filter element to obtain multiplied values;
- and
- processing, by the computation core circuit, subsets of multiplied values to obtain output values.
An image signal processor, comprising:
- an input buffer circuit configured to receive and store data values of a plurality of channels of input data in an interleaved manner, the plurality of channels of input data including at least a first channel of input data and a second channel of input data interleaved with the first channel of input data;
- a datapath switch circuit configured to retrieve, from the input buffer circuit, data values of the first channel of input data and skip data values of the second channel of input data in a cycle, and retrieve the data values of the second channel of input data and skip the data values of the first channel of input data in another cycle;
- a filter switch circuit configured to retrieve filter elements of at least one kernel for performing a convolution operation;
- and
- a computation core circuit configured to:
- receive the at least subset of data values from the datapath switch and the retrieved filter elements from the filter switch,
- multiply each of the data values with a corresponding filter element to obtain multiplied values, and
- process subsets of multiplied values to obtain output values.
Dependencies are inferred from the claim text.