# US 12,112,265 B2 · Architecture for running convolutional networks on memory and mips constrained embedded devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,112,265 B2
- Document ID: US12112265B2
- Status: Granted
- Kind: B2
- Application number: 17/127,560
- Filed: 2020-12-18
- Published: 2022-06-23
- Granted: 2024-10-08
- Earliest claimed: 2020-12-18
- Assignees: Analog Devices International Unlimited Company
- Applicants: Analog Devices International Unlimited Company
- Inventors: Raka Singh; Neeraj Pai; Swastik Mahapatra; Anil M Sripadarao
- Web page: https://patentgrep.com/patent/US12112265B2

## CPC

- G06F 12/0875: with dedicated cache, e.g. instruction or stack
- G06N 3/08: Learning methods
- G06F 7/483: Computations with numbers represented by a non-linear combination of denominational numbers, e.g. rational numbers, logarithmic number system or floating-point numbers {(G06F7/4806, G06F7/4824, G06F7/49, G06F7/491, G06F7/544 take precedence)}
- G06F 12/0802: Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06N 3/02: Neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06F 2212/1016: Performance improvement
- G06F 2212/454: Vector or matrix data
- G06F 2212/60: Details of cache memory
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

This disclosure describes techniques to perform convolutional neural networks (CNNs) on embedded devices. The techniques include operations comprising: accessing DNN information including definition of layers and weights of the DNN; obtaining cache or memory information for one or more cache or memory levels of the resource constrained embedded device; and configuring the DNN to be loaded onto the one or more cache or memory levels of the resource constrained embedded device based on the cache or memory information and the DNN information.

## Claims

### Claim 1

A method for configuring a deep neural network (DNN) to run on a resource constrained embedded device, the method comprising:
accessing DNN information including definition of layers and weights of the DNN;
obtaining cache or memory information for one or more cache or memory levels of the resource constrained embedded device;
configuring the DNN to be loaded onto the one or more cache or memory levels of the resource constrained embedded device based on the cache or memory information and the DNN information;
adjusting one or more weights of the DNN by a division factor;
storing, in a configuration file in association with each of the one or more weights, a correction factor corresponding to the division factor used to adjust the one or more weights of the DNN;
selecting, based on the cache or memory information, between a plurality of DNN processing schemes for loading the DNN information and data onto the resource constrained embedded device, wherein a first of the plurality of DNN processing schemes causes a sub-portion of one of the layers interleaved with the weights and data to be loaded onto a single cache or memory level of the one or more cache or memory levels, and wherein a second of the plurality of DNN processing schemes causes a complete channel corresponding to the layers and the weights and data to be loaded onto the single cache or memory level, wherein selecting the first of the plurality of DNN processing schemes comprises:
computing a first size corresponding to a first subset of rows and columns of the data across all of the layers;
determining that the first size fits within a level 1 cache of the resource constrained embedded device;
causing a second subset of rows and columns of the data across all of the layers to be read into the level 1 cache while the first subset is being processed; and
processing the second subset of the rows and columns based on an output of processing the first subset of the rows and columns, wherein one extra row of a set of data is retrieved from a level 3 cache into the level 1 cache and makes use of two rows that are repeated for a next iteration;
selecting, based on the layers of the DNN, a convolution technique from a plurality of convolution techniques;
processing a given set of data based on the DNN by applying the adjusted one or more weights comprising one or more fractional weights to the given set of data to produce output weighted data;
retrieving the configuration file to obtain the correction factor corresponding to the division factor used to adjust the one or more weights of the DNN; and
multiplying the output weighted data by the correction factor to recover a result corresponding to one or more original weight values of the DNN.

### Claim 2 (depends on 1)

The method of claim 1, wherein the weights of the DNN are floating point weights, further comprising converting the floating point weights to variable length quantized fixed point weights based on a processing capability of the resource constrained embedded device, wherein the one or more weights of the DNN are quantized using one factor and the set of data is quantized using another factor.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
computing a statistical metric of a range of the floating point weights; and
determining a quantization factor for converting the floating point weights to the variable quantized fixed point weights based on the statistical metric.

### Claim 4 (depends on 3)

The method of claim 3, further comprising:
identifying a floating point weight that includes a fractional component and integer component; and
dividing the identified floating point weight by an integer to remove the integer component to improve rounding operations.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
generating the configuration file that merges a layer and model graph structure of the DNN and the weights per each of the layers of the DNN; and
encrypting the configuration file.

### Claim 6 (depends on 5)

The method of claim 5, wherein accessing the DNN information comprising obtaining the configuration file and decrypting the configuration file.

### Claim 7 (depends on 1)

The method of claim 1, wherein the cache information comprises size information of a level 1 cache and level 2 cache of the resource constrained embedded device; and
wherein the memory information comprises a level 3 cache associated with the resource constrained embedded device.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first and second subsets of the rows are stored in a circular buffer.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
in response to determining that the first size exceeds a size of the level 1 cache, computing a second size corresponding to an entire row for a subset of columns of the data across a first layer of the layers; and
in response to determining that the second size fits within a level 1 cache of the resource constrained embedded device, selecting the second of the plurality of DNN processing schemes.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
storing in the level 1 cache an intermediate output of processing an entire row for the subset of columns of the data across the first layer of the layers; and
causing the entire row for the subset of columns of the data across a second layer of the layers to be combined with the intermediate output.

### Claim 11

A system for configuring a deep neural network (DNN) to run on a resource constrained embedded device, the system comprising:
at least one processor configured to perform operations comprising:
accessing DNN information including definition of layers and weights of the DNN;
obtaining cache or memory information for one or more cache or memory levels of the resource constrained embedded device;
configuring the DNN to be loaded onto the one or more cache or memory levels of the resource constrained embedded device based on the cache or memory information and the DNN information;
adjusting one or more weights of the DNN by a division factor;
storing, in a configuration file in association with each of the one or more weights, a correction factor corresponding to the division factor used to adjust the one or more weights of the DNN;
selecting, based on the cache or memory information, between a plurality of DNN processing schemes for loading the DNN information and data onto the resource constrained embedded device, wherein a first of the plurality of DNN processing schemes causes a sub-portion of one of the layers interleaved with the weights and data to be loaded onto a single cache or memory level of the one or more cache or memory levels, and wherein a second of the plurality of DNN processing schemes causes a complete channel corresponding to the layers and the weights and data to be loaded onto the single cache or memory level, wherein selecting the first of the plurality of DNN processing schemes comprises:
computing a first size corresponding to a first subset of rows and columns of the data across all of the layers;
determining that the first size fits within a level 1 cache of the resource constrained embedded device;
causing a second subset of rows and columns of the data across all of the layers to be read into the level 1 cache while the first subset is being processed; and
processing the second subset of the rows and columns based on an output of processing the first subset of the rows and columns, wherein one extra row of a set of data is retrieved from a level 3 cache into the level 1 cache and makes use of two rows that are repeated for a next iteration;
selecting, based on the layers of the DNN, a convolution technique from a plurality of convolution techniques;
processing a given set of data based on the DNN by applying the adjusted one or more weights comprising one or more fractional weights to the given set of data to produce output weighted data;
retrieving the configuration file to obtain the correction factor corresponding to the division factor used to adjust the one or more weights of the DNN; and
multiplying the output weighted data by the correction factor to recover a result corresponding to one or more original weight values of the DNN.

### Claim 12 (depends on 11)

The system of claim 11, wherein the weights of the DNN are floating point weights, further comprising converting the floating point weights to quantized fixed point weights based on a processing capability of the resource constrained embedded device.

### Claim 13 (depends on 12)

The system of claim 12, wherein the operations further comprise:
computing a statistical metric of a range of the floating point weights; and
determining a quantization factor for converting the floating point weights to the quantized fixed point weights based on the statistical metric.

### Claim 14 (depends on 13)

The system of claim 13, wherein the operations further comprise:
identifying a floating point weight that includes a fractional component and integer component; and
dividing the identified floating point weight by an integer to remove the integer component to improve rounding operations.

### Claim 15

A non-transitory computer readable medium storing non-transitory computer readable instructions to configure one or more processors to perform operations for configuring a deep neural network (DNN) to run on a resource constrained embedded device, the operations comprising:
accessing DNN information including definition of layers and weights of the DNN;
obtaining cache or memory information for one or more cache or memory levels of the resource constrained embedded device;
configuring the DNN to be loaded onto the one or more cache or memory levels of the resource constrained embedded device based on the cache or memory information and the DNN information;
adjusting one or more weights of the DNN by a division factor;
storing, in a configuration file in association with each of the one or more weights, a correction factor corresponding to the division factor used to adjust the one or more weights of the DNN;
selecting, based on the cache or memory information, between a plurality of DNN processing schemes for loading the DNN information and data onto the resource constrained embedded device, wherein a first of the plurality of DNN processing schemes causes a sub-portion of one of the layers interleaved with the weights and data to be loaded onto a single cache or memory level of the one or more cache or memory levels, and wherein a second of the plurality of DNN processing schemes causes a complete channel corresponding to the layers and the weights and data to be loaded onto the single cache or memory level, wherein selecting the first of the plurality of DNN processing schemes comprises:
computing a first size corresponding to a first subset of rows and columns of the data across all of the layers;
determining that the first size fits within a level 1 cache of the resource constrained embedded device;
causing a second subset of rows and columns of the data across all of the layers to be read into the level 1 cache while the first subset is being processed; and
processing the second subset of the rows and columns based on an output of processing the first subset of the rows and columns, wherein one extra row of a set of data is retrieved from a level 3 cache into the level 1 cache and makes use of two rows that are repeated for a next iteration;
selecting, based on the layers of the DNN, a convolution technique from a plurality of convolution techniques;
processing a given set of data based on the DNN by applying the adjusted one or more weights comprising one or more fractional weights to the given set of data to produce output weighted data;
retrieving the configuration file to obtain the correction factor corresponding to the division factor used to adjust the one or more weights of the DNN; and
multiplying the output weighted data by the correction factor to recover a result corresponding to one or more original weight values of the DNN.

### Claim 16 (depends on 15)

The non-transitory computer readable medium of claim 15, wherein the weights of the DNN are floating point weights, further comprising converting the floating point weights to quantized fixed point weights based on a processing capability of the resource constrained embedded device.

### Claim 17 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the operations further comprise:
computing a statistical metric of a range of the floating point weights; and
determining a quantization factor for converting the floating point weights to the quantized fixed point weights based on the statistical metric.

### Claim 18 (depends on 17)

The non-transitory computer readable medium of claim 17, wherein the operations further comprise:
identifying a floating point weight that includes a fractional component and integer component; and
dividing the identified floating point weight by an integer to remove the integer component to improve rounding operations.