# US 11,907,830 B2 · Neural network architecture using control logic determining convolution operation sequence > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,830 B2 - Document ID: US11907830B2 - Status: Granted - Kind: B2 - Application number: 18/093,768 - Filed: 2023-01-05 - Published: 2023-05-18 - Granted: 2024-02-20 - Earliest claimed: 2017-11-06 - Assignees: Imagination Technologies Limited - Applicants: Imagination Technologies Limited - Inventors: Christopher Martin - Web page: https://patentgrep.com/patent/US11907830B2 ## CPC - G06N 3/063: using electronic means - G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)} - G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)} - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/048: Activation functions - G06N 3/0495: Quantised networks; Sparse networks; Compressed networks - G06N 3/045: Combinations of networks - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Hardware for implementing a Deep Neural Network (DNN) having a convolution layer. A plurality of convolution engines each perform a convolution operation by applying a filter to a data window. Each of the plurality of convolution engines includes multiplication logic that combines a weight of a filter with a respective data value of a data window; control logic that receives configuration information identifying a set of filters for operation on a set of data windows at the plurality of convolution engines; determines a sequence of convolution operations for evaluation at the multiplication logic; requests weights and data values for at least partially applying a filter to a data window; and causes the multiplication logic to combine the weights with their respective data values. Accumulation logic accumulates the results of a plurality of combinations performed by the multiplication logic to form an output for a convolution operation of the determined sequence. ## Claims ### Claim 1 Hardware for implementing a Deep Neural Network (DNN), the hardware comprising a plurality of convolution engines each operable to perform a convolution operation by applying a filter to a data window, each filter comprising a set of weights for combination with respective data values of the data window, and each of the plurality of convolution engines comprising: multiplication logic operable to combine a filter weight with a respective data window value; control logic configured to: receive configuration information identifying a set of filters for combination with a set of data windows, determine, using the configuration information, a sequence of convolution operations, and in dependence on the determined sequence of convolution operations, cause the multiplication logic to at least partially combine the set of filters with respective data windows of the set of data windows; and accumulation logic configured to accumulate results of a plurality of combinations performed by the multiplication logic so as to form an output for a convolution operation of the determined sequence. ### Claim 2 (depends on 1) Hardware as claimed in claim 1, wherein the control logic is further configured to: in accordance with the determined sequence of convolution operations, request filter weights and data windows values, wherein the partially combining the set of filters with respective data windows comprises combining the filter weights with their respective data window values. ### Claim 3 (depends on 1) Hardware as claimed in claim 1, wherein each of the plurality of convolution engines is arranged to independently perform a different sequence of convolution operations such that collectively the convolution engines operate the set of filters on each data window of the set of data windows. ### Claim 4 (depends on 1) Hardware as claimed in claim 1, wherein each of the plurality of convolution engines is configured to independently request filter weights and data window values for a next convolution operation and cause its multiplication logic to commence that next convolution operation independently of whether or not other convolution engines of the plurality have completed their current convolution operations. ### Claim 5 (depends on 1) Hardware as claimed in claim 1, wherein the configuration information comprises the number of filters in the set of filters and the number of data windows in the set of data windows, and the control logic is configured to determine a sequence of convolution operations using the configuration information and the number of convolution engines arranged to apply the set of filters to the set of data windows. ### Claim 6 (depends on 1) Hardware as claimed in claim 1, wherein the hardware is arranged to perform a predetermined sequence of convolution operations, each convolution operation being identified by an index according to which each next convolution operation is identified by incrementing the index, and on completing a convolution operation each convolution engine is configured to determine its next convolution operation by incrementing the index of its current convolution operation by a step value derived from the number of convolution engines such that the sequence of convolution operations performed at each convolution engine is also predetermined. ### Claim 7 (depends on 6) Hardware as claimed in claim 6, wherein the step value is the number of convolution engines. ### Claim 8 (depends on 6) Hardware as claimed in claim 6, wherein the set of filters are arranged to be performed in a fixed sequence over each of the set of data windows so as to form the predetermined sequence of convolution operations, and the control logic is configured to determine filter weights and data window values for each next convolution operation by determining the filter and data window for that next convolution operation according to the predetermined sequence. ### Claim 9 (depends on 1) Hardware as claimed in claim 1, wherein the control logic is configured to request next filter weights and data window values prior to the multiplication logic completing its combination of current filter weights and data window values. ### Claim 10 (depends on 1) Hardware as claimed in claim 1, wherein the configuration information is received at each convolution engine at the start of a hardware pass, the set of filters for application to the set of data windows identified by the configuration information being the complete set of filters and complete set of data windows for evaluation by the plurality of convolution engines of the hardware in that hardware pass. ### Claim 11 (depends on 1) Hardware as claimed in claim 1, wherein each convolution engine is configured to identify zero filter weights in the requested filter weights and, for each filter weight and its respective data window value, cause the multiplication logic to combine the filter weight with its respective data window value only if the filter weight is non-zero. ### Claim 12 (depends on 11) Hardware as claimed in claim 11, wherein each convolution engine is further configured to identify zero data window values in the requested data window values and, for each filter weight and its respective data window value, cause the multiplication logic to combine the data window value with its respective filter weight only if the filter weight and data window value are non-zero. ### Claim 13 (depends on 1) Hardware as claimed in claim 1, further comprising input data logic for controlling one or more input buffers arranged to store data window values for the data windows required by the plurality of convolution engines, the input data logic being configured to respond to the requests for data window values from each convolution engines, optionally wherein, in each request for data window values, the control logic of each convolution engine is configured to identify the data window to which the data window values relate and the filter to which the filter weights relate, and the input data logic is configured to replace that data window with the next data window according to the predetermined sequence of convolution operations when the final convolution engine using that data window requests the final data window values of that data window and the final filter weights of the final filter according to the predetermined sequence of convolution operations. ### Claim 14 A method for implementing a Deep Neural Network (DNN) in hardware, the hardware comprising a plurality of convolution engines each operable to perform a convolution operation by applying a filter to a data window, and each filter comprising a set of weights for combination with respective data values of the data window, the method comprising, at each of the plurality of convolution engines: receiving configuration information identifying a set of the filters for combination with a set of the data windows; determining, using the configuration information, a sequence of convolution operations; in dependence on the determined sequence of convolution operations, at least partially combining, by multiplication, the set of filters with respective data windows of the set of data windows; and accumulating results of a plurality of the multiplications so as to form an output for a convolution operation of the determined sequence. ### Claim 15 (depends on 14) The method as claimed in claim 14, further comprising causing each of the plurality of convolution engines to independently perform a different sequence of convolution operations such that collectively the convolution engines apply the set of filters to each data window of the set of data windows. ### Claim 16 (depends on 14) The method as claimed in claim 14, wherein the requesting performed by each of the plurality of convolution engines comprises independently requesting filter weights and data window values for a next convolution operation and commencing multiplication of filter weights and data window values of that next convolution operation independently of whether or not other convolution engines of the plurality have completed their current convolution operations. ### Claim 17 (depends on 14) The method as claimed in claim 14, wherein the configuration information comprises the number of filters in the set of filters and the number of data windows in the set of data windows, and the determining comprises determining a sequence of convolution operations using the configuration information and the number of convolution engines arranged to apply the set of filters to the set of data windows. ### Claim 18 (depends on 14) The method as claimed in claim 14, wherein the convolution operations are arranged in a predetermined sequence and each convolution operation is identified by an index according to which each next convolution operation is identified by incrementing the index, and the determining comprises, on the convolution engine completing its convolution operation, determining its next convolution operation by incrementing the index of its current convolution operation by a step value derived from the number of convolution engines such that the sequence of convolution operations performed at each convolution engine is also predetermined. ### Claim 19 (depends on 1) Hardware as set forth in claim 1, wherein the hardware is embodied on an integrated circuit. ### Claim 20 A non-transitory computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to perform a method for implementing a Deep Neural Network (DNN) in hardware, the hardware comprising a plurality of convolution engines each operable to perform a convolution operation by applying a filter to a data window, and each filter comprising a set of weights for combination with respective data values of the data window, the method comprising, at each of the plurality of convolution engines: receiving configuration information identifying a set of the filters for combination with a set of the data windows; determining, using the configuration information, a sequence of convolution operations; in dependence on the determined sequence of convolution operations, at least partially combining, by multiplication, the set of filters with respective data windows of the set of data windows; and accumulating results of a plurality of the multiplications so as to form an output for a convolution operation of the determined sequence.