# US 2024/0202500 A1 · Acceleration of 2d dilated convolution for efficient analytics

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202500 A1
- Document ID: US20240202500A1
- Status: Published application
- Kind: A1
- Application number: 18/067,089
- Filed: 2022-12-16
- Published: 2024-06-20
- Earliest claimed: 2022-12-16
- Assignees: Texas Instruments Incorporated
- Applicants: Texas Instruments Incorporated
- Inventors: Varun Tripathi; William Leven; Pramod Swami
- Web page: https://patentgrep.com/patent/US20240202500A1

## CPC

- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/045: Combinations of networks
- G06N 3/063: using electronic means
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed herein are improved systems and methods for accelerated 2D dilated convolution. A processor determines an offset based on a dilation factor of the 2D dilated convolution. The processor selects rows of data from the 2D input in phases based on the offset and loads an input feature panel without overwriting data that has not yet been consumed by the 2D dilated convolution processor. As the 2D dilated convolution processor performs the convolution iterations, the processor continues to load additional data for the convolution. As the convolution iterations are completed, the processor spaces result of the 2D dilated convolution into a matrix such that results of each phase are spaced based on the offset.

## Claims

### Claim 1

A method comprising:
determining, by a processor, an offset based on a dilation factor of a two-dimensional (2D) dilated convolution associated with 2D input;
selecting, by the processor, rows of data from the 2D input for the 2D dilated convolution in phases based on the offset; and
spacing, by the processor, results of the 2D dilated convolution at each of the phases based on the offset.

### Claim 2 (depends on 1)

The method of claim 1, wherein determining the offset comprises:
selecting the dilation factor as the offset.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
supplying, by the processor, a filter vector to the 2D dilated convolution.

### Claim 4 (depends on 1)

The method of claim 1, wherein selecting the rows of data from the 2D input for the 2D dilated convolution in phases based on the offset comprises:
in each phase of a number of phases, wherein the number of phases is based on the dilation factor:
loading data into an input feature panel of the 2D dilated convolution based on applying a dilated filter comprising filter coefficients to a set of rows of an input feature map comprising the data from the 2D input, wherein the set of rows is selected based on the offset.

### Claim 5 (depends on 4)

The method of claim 4, further comprising, in each phase of the number of phases:
supplying, by the processor, a filter vector to the 2D dilated convolution, wherein the filter vector comprises the filter coefficients ordered based on the offset and the loading the data.

### Claim 6 (depends on 4)

The method of claim 4, further comprising, in each phase of the number of phases:
shifting, by the processor, the dilated filter down a number of rows of the input feature map, wherein the number of rows is equal to the offset, and wherein the loading the data into the input feature panel comprises loading the data to implement vertical reuse; and
repeating the loading the data until the dilated filter has been shifted to an end of the input feature map.

### Claim 7 (depends on 1)

The method of claim 1, wherein spacing the results of the 2D dilated convolution at each of the phases based on the offset comprises:
storing the results for each phase in rows of a convolution result matrix starting with an uppermost unfilled row and continuing in rows separated by the offset.

### Claim 8

A system comprising:
a hardware accelerator including:
an input feature panel memory,
a convolutional result memory, and
a filter vector memory;
a controller; and
a memory having stored thereon instructions that, upon execution by the controller, cause the controller to:
receive an input feature map,
determine an offset based on a dilation factor,
implement a loading plan to carry out more than one phase of 2-dimensional (2D) dilated convolution based on the dilation factor, each phase including:
applying a dilated filter to the input feature map to load values into the input feature panel memory nonsequentially based on the offset for a convolution iteration; and
instructing the hardware accelerator to perform the convolution iteration, and
implement an output plan including:
retrieving values from the convolutional result memory on completion of the convolution iteration; and
storing the values in a row of a convolution result matrix based on the offset.

### Claim 9 (depends on 8)

The system of claim 8, further comprising:
a camera, wherein the memory has stored thereon further instructions that, upon execution by the controller, cause the controller to:
access an image captured by the camera; and
convert the image into an input feature map.

### Claim 10 (depends on 8)

The system of claim 8, wherein the input feature map is a matrix comprising one or more columns and one or more rows.

### Claim 11 (depends on 10)

The system of claim 10, wherein the applying the dilated filter to the input feature map comprises shifting the dilated filter horizontally across the input feature map by a factor of 1 until the dilated filter has been shifted to an end of the row of the input feature map.

### Claim 12 (depends on 8)

The system of claim 8, wherein the values loaded from the input feature map into the input feature panel memory comprise values from rows of the input feature map to which the dilated filter is newly applied.

### Claim 13 (depends on 8)

The system of claim 8, wherein the dilated filter comprises filter coefficients separated by the dilation factor.

### Claim 14 (depends on 13)

The system of claim 13, wherein implementing the loading plan further comprises: loading the filter coefficients into the filter vector memory based on the offset for the convolution iteration.

### Claim 15

A method of executing dilated convolution, comprising:
receiving, by a controller, an input feature map;
executing a phase of a number of phases of dilated convolution of the input feature map, wherein the number of phases is based on a dilation factor of the dilated convolution, and wherein the executing the phase comprises:
selecting, by the controller, a starting row of the input feature map for the phase,
convolving an input feature panel with a filter vector wherein the convolving comprises:
applying, by the controller, a dilated filter to the input feature map, wherein the dilated filter comprises a set of filter coefficients separated by the dilation factor;
loading, by the controller, a set of values into an input feature panel based on applying the dilated filter to a set of rows of the input feature map to implement vertical reuse;
accessing, by a convolution processor, the filter vector comprising the set of filter coefficients;
convolving, by the convolution processor, the filter vector with the input feature panel to generate an output;
assigning, by the controller, the output to a selected row of a convolution result matrix wherein the selected row is selected based on the dilation factor and the phase; and
shifting, by the controller, the dilated filter down the input feature map based on the dilation factor, and
repeating the convolving until the dilated filter has been shifted to an end of the input feature map; and
repeating the executing the phase for each phase of the number of phases.

### Claim 16 (depends on 15)

The method of claim 15, wherein to implement vertical reuse, the set of values comprises of values from rows from the input feature map to which the dilated filter is newly applied.

### Claim 17 (depends on 15)

The method of claim 15, wherein selecting the starting row comprises selecting an uppermost row of the input feature map that has not yet had dilated filter applied.

### Claim 18 (depends on 15)

The method of claim 15, wherein loading the set of values into the input feature panel comprises shifting the dilated filter horizontally across the input feature map by a factor of one, until the dilated filter has been shifted to an end of the row of the input feature map.

### Claim 19 (depends on 15)

The method of claim 15, wherein accessing the filter vector further comprises loading, by the controller, the filter coefficients into the filter vector based on the vertical reuse.

### Claim 20 (depends on 15)

The method of claim 15, wherein accessing the filter vector comprises:
accessing a memory storing a number of filter vectors wherein each of the number of filter vectors comprises the filter coefficients in a different order; and
selecting the filter vector from the number of filter vectors based on the vertical reuse.
