# US 10,482,337 B2 · Accelerating convolutional neural network computation throughput > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,482,337 B2 - Document ID: US10482337B2 - Status: Granted - Kind: B2 - Application number: 15/719,829 - Filed: 2017-09-29 - Published: 2019-04-04 - Granted: 2019-11-19 - Earliest claimed: 2017-09-29 - Assignees: Infineon Technologies AG - Applicants: Infineon Technologies AG - Inventors: Prakash Kalanjeri Balasubramanian - Web page: https://patentgrep.com/patent/US10482337B2 ## CPC - G06N 3/063: using electronic means - G06F 3/0625: {Power saving in storage systems} - G06F 3/0656: {Data buffering arrangements} - G06F 3/0659: {Command handling arrangements, e.g. command buffers, queues, command scheduling} - G06F 3/0685: {Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06F 18/24: Classification techniques - G06F 18/2414: {Smoothing the distance, e.g. radial basis function networks [RBFN]} - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/048: Activation functions - G06V 10/82: using neural networks - G06V 10/95: {structured as a network, e.g. client-server architectures} - G06V 20/58: Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads - G06V 20/582: {of traffic signs} - G06V 30/19173: {Classification techniques} - G06V 30/194: References adjustable by an adaptive method, e.g. learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Convolutional neural network (CNN) components can operate to provide various speed-ups to improve upon or operate as part of an artificial neural network (ANN). A convolution component performs convolution operations that extract data from one or more images, and provides the data to one or more rectified linear units (RELUs). The RELUs are configured to generate non-linear convolution output data. A pooling component generates pooling outputs in parallel with the convolution operations via a pipelining process based on a pooling window for a subset of the non-linear convolution output data. A fully connected (FC) component configured to form an artificial neural network (ANN) that provides ANN outputs based on the pooling outputs and enables a recognition of a pattern in the one or more images based on the ANN outputs. Layers of the FC component are also able to operate in parallel in another pipelining process. ## Claims ### Claim 1 A system for a convolutional neural network (CNN) comprising: a convolution component configured to perform convolution operations that extract data from one or more images, and provide the data to one or more rectified linear units (RELUs) configured to generate non-linear convolution output data; a pooling component configured to generate pooling outputs via a pipelining process in parallel with the convolution operations based on a pooling window for a subset of the non-linear convolution output data; an observation unit component comprising one or more first tables corresponding to the pipelining process of the pooling component with the convolution component; a transaction watch component configured to detect a write operation to the one or more first tables by the convolution component and trigger a statistical operation of the pooling component in response to the convolution component performing the write operation to the one or more first tables, and generate a plurality of outputs from the one or more first tables to the pooling component; and a fully connected (FC) component configured to form an artificial neural network (ANN) that provides ANN outputs based on the pooling outputs and enable a recognition of a pattern in the one or more images based on the ANN outputs. ### Claim 2 (depends on 1) The system of claim 1, wherein the pooling component is further configured to determine an availability of the pooling window, and process a plurality of pooling windows that correspond to different subsets of the non-linear convolution output data. ### Claim 3 (depends on 1) The system of claim 1, wherein the FC component comprises a first layer of neurons coupled to a succeeding layer of neurons, wherein neurons of the first layer and the succeeding layer comprise a net function node configured to provide a net function output and an output function node configured to generate an output function output from the net function output, respectively. ### Claim 4 (depends on 3) The system of claim 3, wherein the first layer of neurons comprises first neurons having a first set of same net function inputs, respectively, that receive outputs from at least one preceding layer comprising at least one of: the pooling outputs, the data from the convolution component, or the non-linear convolution output data, and the succeeding layer of neurons comprises second neurons having a second set of same net function inputs that receive same outputs from the first layer of neurons, respectively. ### Claim 5 (depends on 3) The system of claim 3, wherein the net function node is configured to perform a dot product multiplication of inputs and weights associated with the inputs, respectively, and a sum of corresponding results, and the output function node is configured to execute an activation function on the net function output to provide an output of an FC component, wherein the output function node of a first neuron operates in parallel with the net function node of a second neuron via another pipelining process. ### Claim 6 (depends on 1) The system of claim 1, wherein the pooling component is further configured to generate a reduction in a volume of the non-linear convolution output data, and process the non-linear convolution output data into scaled invariant data as the pooling outputs to be provided to the FC component. ### Claim 7 (depends on 1) The system of claim 1, wherein the plurality of outputs comprise at least one of: a pooling input address associated with a convolution watch address of the one or more first tables identifying markers associated with the pooling window, the trigger to the operations of the pooling component, a pooling output address pointing to an address in the one or more first tables with a result of the statistical operation, or a convolution layer identifier (ID) associating the one or more first tables to the pooling component. ### Claim 8 (depends on 1) The system of claim 1, wherein a second table associated with the FC layer comprises at least one of: a neuron net function output address corresponding to a net function output of a net function node of the FC component, a neuron activation output address corresponding to an ANN output of an output function node, and an activation function that is executed by the output function node. ### Claim 9 An apparatus for a convolutional neural network (CNN), comprising: one or more processors, coupled to a memory that includes instructions to execute operations of the one or more processors, configured to: extract data features that comprise convolutional data sets via one or more filter kernels from one or more images; transform the convolutional data sets into sets of nonlinear output data; generate pooling outputs from pooling operations on the sets of nonlinear output data via a pipelining process that operates in parallel with at least one of an extraction of the data features or a transformation of the convolutional data sets, in response to determining that a first set of nonlinear data among the sets of nonlinear output data or a first set of convolutional data of the convolutional data sets is available; and populate a table comprising entries for a neuron net function output address corresponding to a net function output, a neuron activation function output address corresponding to an output of a neuron, and an activation function executed by an output function of the neuron to generate the output of the neuron based on the net function output. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the one or more processors are further configured to extract the data features by performing sliding convolutions with one or more convolution windows sliding over portions of the one or more images to generate feature maps of the one or more images and determine a likelihood of a presence of a data feature corresponding to the one or more convolution windows. ### Claim 11 (depends on 9) The apparatus of claim 9, wherein the one or more processors are further configured to: extract a second set of convolutional data of the convolutional data sets from the one or more images via a convolution operation in parallel with a pooling operation that processes the first set of convolutional data of the convolutional data sets into a scaled invariant set of convolutional data based on a pooling window. ### Claim 12 (depends on 9) The apparatus of claim 9, wherein the one or more processors are further configured to: process the pooling outputs via a fully connected (FC) component by classifying the pooling outputs corresponding to the sets of nonlinear data into categories. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein the one or more processors are further configured to: perform a net function comprising a dot product of inputs and weights and a sum of weighted inputs from the pooling outputs at a first FC component to generate a neuron net function output; and perform an output function on the neuron net function output at the first FC component to generate a neuron activation function output in parallel with performing another net function comprising another dot product and another sum of the inputs from the pooling outputs at a succeeding FC component to generate a second neuron net function output. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the first FC component and the succeeding FC component comprise data flow processes that operate without fetching an instruction from the memory and in response to input data from the pooling outputs or a preceding FC component being available at neuron inputs. ### Claim 15 A method for a convolutional neural network (CNN), comprising: performing, via one or more processors, convolution operations that extract data from one or more images; providing, via the one or more processors, the data to one or more rectified linear units (RELUs) configured to generate non-linear convolution output data; generating, via the one or more processors, pooling outputs in parallel with the convolution operations based on a pooling window for a subset of the non-linear convolution output data; detecting, via the one or more processors, a write to one or more tables corresponding to pipelining processes for the pooling outputs in parallel with the convolution operations to trigger a statistical operation of the pipelining processes for generation of the pooling outputs; and providing, via the one or more processors, artificial neural network (ANN) outputs of an ANN based on the pooling outputs. ### Claim 16 (depends on 15) The method of claim 15, further comprising: generating a determination of an availability of the pooling window, and process a plurality of pooling windows that correspond to different subsets of the non-linear convolution output data. ### Claim 17 (depends on 15) The method of claim 15, further comprising: providing, in a first fully connected (FC) layer of the ANN, first net function outputs of first net function nodes to first output function nodes; and providing first output function outputs of the first output function nodes to second net function nodes of a succeeding FC layer that is configured to generate the ANN outputs. ### Claim 18 (depends on 17) The method of claim 17, further comprising: determining a completion of a first net function node of the first net function nodes; and initiating, in response to the completion, a second net function of the second net function nodes in parallel with a first output network node generating an output function of the first FC layer.