# US 11,443,176 B2 · Acceleration of convolutional neural networks on analog arrays > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,443,176 B2 - Document ID: US11443176B2 - Status: Granted - Kind: B2 - Application number: 16/362,031 - Filed: 2019-03-22 - Published: 2019-11-21 - Granted: 2022-09-13 - Earliest claimed: 2018-05-17 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Malte Rasch; Tayfun Gokmen; Mattia Rigotti; Wilfried Haensch - Web page: https://patentgrep.com/patent/US11443176B2 ## CPC - G06N 3/065: Analogue means - G06N 3/045: Combinations of networks - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/084: Backpropagation, e.g. using gradient descent - G06N 3/09: Supervised learning - G06T 7/11: Region-based segmentation - G06T 2207/20021: Dividing image into blocks, subimages or windows - G06T 2207/20081: Training; Learning - G06T 2207/20084: Artificial neural networks [ANN] - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract Mechanisms are provided for acceleration of convolutional neural networks on analog arrays. Input ports receive image signals from frames in an input image. Input memory arrays store the image signals received from the input ports into a respective input memory location to create a plurality of image sub-regions in input memory arrays. A distributor associated each of a set of analog array tiles in an analog array to a part of image sub-regions of the input memory arrays, so that one or more of a set of analog memory components is associated with the image signals in a distribution order to create a respective output signal. An assembler stores each of the respective output signals into one of a set of memory outputs in an output order that is determined by the distribution order. ## Claims ### Claim 1 A computer system for acceleration of convolutional neural networks on analog arrays comprising: a processor; and a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to: receive, via one or more input ports, image signals from one or more frames in an input image; store, in one or more input memory arrays, the image signals received from the one or more input ports into a respective input memory location to create a plurality of image sub-regions in the one or more input memory arrays, the image sub-regions being in an image sub-region order in the one or more input memory arrays; associate, via a distributor, each of a set of analog array tiles in an analog array to a part of one or more image sub-regions of the one or more input memory arrays, so that one or more of a set of analog memory components is associated with one or more of the image signals in a distribution order to create a respective output signal, each of the set of analog array tiles associated with a respective analog array tile in the set of analog array tiles in parallel with other analog array tiles of the set of analog array tiles, each analog array tile having a plurality of analog memory components, one or more of the set of analog memory components having weighting factors; and store, via an assembler, each of the respective output signals into one of a set of memory outputs in an output order that is determined by the distribution order, the set of memory outputs being part of one or more output memory arrays with each memory output associated with a part of an output image. ### Claim 2 (depends on 1) The system of claim 1, wherein one or more of the plurality of image sub-regions is an n×n pixel region of the input image. ### Claim 3 (depends on 1) The system of claim 1, wherein the image sub-region order includes having one or more image sub-regions of the plurality of image sub-regions overlap one or more other image sub-regions. ### Claim 4 (depends on 1) The system of claim 1, wherein the one or more image sub-regions are stored in one linear element of the one or more input memory arrays. ### Claim 5 (depends on 1) The system of claim 1, wherein the analog array is one or more of: a discrete analog array, a cross-bar switch analog array, a configurable analog array, an array of analog switches, or a resistive process unit (RPU). ### Claim 6 (depends on 1) The system of claim 1, wherein the weighting factors in the analog array dynamically change as a result of the distribution by back propagation. ### Claim 7 (depends on 1) The system of claim 1, wherein a size, a shape, or a number of the set of analog array tiles in the analog array is re-configurable. ### Claim 8 (depends on 1) The system of claim 1, wherein the output order of the assembler is a row position in the set of memory outputs that corresponds to the row position on a memory input. ### Claim 9 (depends on 1) The system of claim 1, wherein multiple analog array tiles are randomly associated with an image sub-regions. ### Claim 10 (depends on 1) The system of claim 1, wherein the distribution order is one or more of random tiling, perforated convolution, image-based tiling, or alternate tiling. ### Claim 11 A method, in a data processing system, for acceleration of convolutional neural networks on analog arrays, the method comprising: receiving, via one or more input ports, image signals from one or more frames in an input image; storing, in one or more input memory arrays, the image signals received from the one or more input ports into a respective input memory location to create a plurality of image sub-regions in the one or more input memory arrays, the image sub-regions being in an image sub-region order in the one or more input memory arrays; associating, via a distributor, each of a set of analog array tiles in an analog array to a part of one or more image sub-regions of the one or more input memory arrays, so that one or more of a set of analog memory components is associated with one or more of the image signals in a distribution order to create a respective output signal, each of the set of analog array tiles associated with a respective analog array tile in the set of analog array tiles in parallel with other analog array tiles of the set of analog array tiles, each analog array tile having a plurality of analog memory components, one or more of the set of analog memory components having weighting factors; and storing, via an assembler, each of the respective output signals into one of a set of memory outputs in an output order that is determined by the distribution order, the set of memory outputs being part of one or more output memory arrays with each memory output associated with a part of an output image. ### Claim 12 (depends on 11) The method of claim 11, wherein one or more of the plurality of image sub-regions is an n×n pixel region of the input image. ### Claim 13 (depends on 11) The method of claim 11, wherein the image sub-region order includes having one or more image sub-regions of the plurality of image sub-regions overlap one or more other image sub-regions. ### Claim 14 (depends on 11) The method of claim 11, wherein the one or more image sub-regions are stored in one linear element of the one or more input memory arrays. ### Claim 15 (depends on 11) The method of claim 11, wherein the analog array is one or more of: a discrete analog array, a cross-bar switch analog array, a configurable analog array, an array of analog switches, or a resistive process unit (RPU). ### Claim 16 (depends on 11) The method of claim 11, wherein a size, a shape, or a number of the set of analog array tiles in the analog array is re-configurable. ### Claim 17 (depends on 11) The method of claim 11, wherein the output order of the assembler is a row position in the set of memory outputs that corresponds to the row position on a memory input. ### Claim 18 (depends on 11) The method of claim 11, wherein multiple analog array tiles are randomly associated with an image sub-regions. ### Claim 19 (depends on 11) The method of claim 11, wherein the distribution order is one or more of random tiling, perforated convolution, image-based tiling, or alternate tiling. ### Claim 20 A computer program product comprising a computer readable storage medium having a computer readable program for acceleration of convolutional neural networks on analog arrays stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to: receive, via one or more input ports, image signals from one or more frames in an input image; store, in one or more input memory arrays, the image signals received from the one or more input ports into a respective input memory location to create a plurality of image sub-regions in the one or more input memory arrays, the image sub-regions being in an image sub-region order in the one or more input memory arrays; associate, via a distributor, each of a set of analog array tiles in an analog array to a part of one or more image sub-regions of the one or more input memory arrays, so that one or more of a set of analog memory components is associated with one or more of the image signals in a distribution order to create a respective output signal, each of the set of analog array tiles associated with a respective analog array tile in the set of analog array tiles in parallel with other analog array tiles of the set of analog array tiles, each analog array tile having a plurality of analog memory components, one or more of the set of analog memory components having weighting factors; and store, via an assembler, each of the respective output signals into one of a set of memory outputs in an output order that is determined by the distribution order, the set of memory outputs being part of one or more output memory arrays with each memory output associated with a part of an output image.