# US 2021/0241070 A1 · Hybrid convolution operation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241070 A1
- Document ID: US20210241070A1
- Status: Published application
- Kind: A1
- Application number: 17/165,648
- Filed: 2021-02-02
- Published: 2021-08-05
- Earliest claimed: 2020-02-03
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Eric Wayne MAHURIN
- Web page: https://patentgrep.com/patent/US20210241070A1

## CPC

- G06N 3/04: Architecture, e.g. interconnection topology
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A device includes one or more processors configured to retrieve a first block of data, the data corresponding to array of values arranged along at least a first dimension and a second dimension, to retrieve at least a portion of a second block of the data, and to perform a first hybrid convolution operation that applies a filter across the first block and at least the portion of the second block to generate output data. The output data includes a first accumulated block and at least a portion of a second accumulated block. The one or more processors are also configured to store the first accumulated block as first output data. The portion of the second block is adjacent to the first block along the first dimension and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension.

## Claims

### Claim 1

A device comprising:
a memory configured to store data corresponding to an array of values arranged along at least a first dimension and a second dimension; and
one or more processors configured to:
retrieve a first block of the data and at least a portion of a second block of the data from the memory;
perform a first hybrid convolution operation that applies a filter across the first block and at least the portion of the second block to generate output data,
wherein the output data includes a first accumulated block and at least a portion of a second accumulated block; and
store the first accumulated block as first output data, wherein the portion of the second block is adjacent to the first block along the first dimension and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension.

### Claim 2 (depends on 1)

The device of claim 1, wherein each of the first block, the second block, the first accumulated block, and the second accumulated block is sized in accordance with a memory access data size that is supported by the memory for memory accesses by the one or more processors.

### Claim 3 (depends on 2)

The device of claim 2, wherein the first hybrid convolution operation uses zero padding along the second dimension without using zero padding along the first dimension to enable convolution using at most two memory accesses to retrieve the first block and the portion of the second block and using at most two memory accesses to store the first accumulated block and the portion of the second accumulated block.

### Claim 4 (depends on 1)

The device of claim 1, wherein:
the first block and the portion of the second block together correspond to N rows and a first M columns of the array, N and M are positive integers, and N is greater than M;
the first accumulated block and the second accumulated block together correspond to M rows and N columns of output data;
the first accumulated block corresponds to M rows and M columns of accumulated values in a first accumulator; and
the portion of the second accumulated block corresponds to M rows and N-M columns of partial accumulated values in a second accumulator.

### Claim 5 (depends on 4)

The device of claim 4, wherein the one or more processors are further configured to access the first block using a first memory access operation and to access the portion of the second block using a second memory access operation, and wherein each of the first memory access operation and the second memory access operation is configured to retrieve, at most, M rows and M columns of the data.

### Claim 6 (depends on 4)

The device of claim 4, wherein the filter includes P rows and P columns of filter values, and wherein P=N−M+1.

### Claim 7 (depends on 4)

The device of claim 4, wherein the one or more processors are further configured to:
retrieve a third block and at least a portion of a fourth block of the data that together correspond to the N rows and a second M columns of the array;
perform a second hybrid convolution operation on the third block and the portion of the fourth block to update the second accumulated block and to generate at least a portion of a third accumulated block that is adjacent to the second accumulated block along the second dimension; and
store the second accumulated block as second output data.

### Claim 8 (depends on 7)

The device of claim 7, wherein the one or more processors are further configured to perform a sequence of hybrid convolution operations including the first hybrid convolution operation and the second hybrid convolution operation in a pipelined manner in which storage of the first accumulated block and performance of the second hybrid convolution operation occur during a same pipeline cycle.

### Claim 9 (depends on 7)

The device of claim 7, wherein the one or more processors are further configured to:
generate the first accumulated block in a first accumulator;
generate the second accumulated block in a second accumulator; and
generate the third accumulated block in the first accumulator after storing the first accumulated block.

### Claim 10 (depends on 1)

The device of claim 1, wherein the one or more processors are configured to convolve the data with the filter in accordance with an image processing operation.

### Claim 11 (depends on 1)

The device of claim 1, wherein the one or more processors are configured to convolve the data with the filter in accordance with a neural network convolutional layer operation.

### Claim 12 (depends on 11)

The device of claim 11, wherein the memory and the one or more processors are integrated into a vehicle.

### Claim 13 (depends on 11)

The device of claim 11, wherein the memory and the one or more processors are integrated into a home automation system.

### Claim 14 (depends on 1)

The device of claim 1, wherein the memory and the one or more processors are integrated into a mobile device.

### Claim 15 (depends on 1)

The device of claim 1, wherein the one or more processors are included in an integrated circuit.

### Claim 16

A method of performing convolution, comprising:
retrieving, at one or more processors, a first block of data and at least a portion of a second block of data from a memory, the data corresponding to an array of values arranged along at least a first dimension and a second dimension;
performing a first hybrid convolution operation that applies a filter across the first block and at least the portion of the second block to generate output data,
wherein the output data includes a first accumulated block and at least a portion of a second accumulated block; and
storing the first accumulated block as first output data, wherein the portion of the second block is adjacent to the first block along the first dimension and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension.

### Claim 17 (depends on 16)

The method of claim 16, wherein each of the first block, the second block, the first accumulated block, and the second accumulated block is sized in accordance with a memory access data size that is supported by the memory for memory accesses by the one or more processors.

### Claim 18 (depends on 16)

The method of claim 16, wherein the first hybrid convolution operation uses zero padding along the second dimension without using zero padding along the first dimension to enable convolution using at most two memory accesses to retrieve the first block and the portion of the second block and using at most two memory accesses to store the first accumulated block and the portion of the second accumulated block.

### Claim 19 (depends on 16)

The method of claim 16, wherein retrieving the first block includes using a first memory access operation, wherein retrieving the portion of the second block includes using a second memory access operation, wherein each of the first memory access operation and the second memory access operation is configured to retrieve, at most, M rows and M columns of the data, wherein M is a positive integer, and wherein each of the first accumulated block and the second accumulated block corresponds to M rows and M columns of output data.

### Claim 20 (depends on 19)

The method of claim 19, further comprising:
retrieving a third block of the data that is adjacent to the first block along the first dimension of the array;
retrieving at least a portion of a fourth block of the data that is adjacent to the portion of the second block along the first dimension of the array;
performing a second hybrid convolution operation on the third block and the portion of the fourth block to update the second accumulated block and to generate at least a portion of a third accumulated block that is adjacent to the second accumulated block along the second dimension; and
storing the second accumulated block as second output data.

### Claim 21 (depends on 20)

The method of claim 20, wherein the first hybrid convolution operation and the second hybrid convolution operation are part of a sequence of hybrid convolution operations that are performed in a pipelined manner in which storage of the first accumulated block and performance of the second hybrid convolution operation occur during a same pipeline cycle.

### Claim 22 (depends on 20)

The method of claim 20, further comprising:
generating the first accumulated block in a first accumulator;
generating the second accumulated block in a second accumulator; and
generating the third accumulated block in the first accumulator after storing the first accumulated block.

### Claim 23 (depends on 16)

The method of claim 16, wherein the one or more processors are configured to convolve the data with the filter in accordance with at least one of an image processing operation or a neural network convolutional layer operation.

### Claim 24

A computer-readable storage device storing instructions that, when executed by one or more processors, cause the one or more processors to:
retrieve a first block of data and at least a portion of a second block of data from a memory, the data corresponding to an array of values arranged along at least a first dimension and a second dimension;
perform a first hybrid convolution operation that applies a filter across the first block and at least the portion of the second block to generate output data,
wherein the output data includes a first accumulated block and at least a portion of a second accumulated block; and
store the first accumulated block as first output data, wherein the portion of the second block is adjacent to the first block along the first dimension and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension.

### Claim 25 (depends on 24)

The computer-readable storage device of claim 24, wherein each of the first block, the second block, the first accumulated block, and the second accumulated block is sized in accordance with a memory access data size that is supported by the memory for memory accesses by the one or more processors.

### Claim 26 (depends on 24)

The computer-readable storage device of claim 24, wherein each of the first block, the second block, the first accumulated block, and the second accumulated block have the same size.

### Claim 27 (depends on 24)

The computer-readable storage device of claim 24, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
retrieve a third block of the data that is adjacent to the first block along the first dimension of the array;
retrieve at least a portion of a fourth block of the data that is adjacent to the portion of the second block along the first dimension of the array;
perform a second hybrid convolution operation on the third block and the portion of the fourth block to update the second accumulated block and to generate at least a portion of a third accumulated block that is adjacent to the second accumulated block along the second dimension; and
store the second accumulated block as second output data.

### Claim 28

An apparatus comprising:
means for retrieving a first block of data and at least a portion of a second block of data, the data corresponding to an array of values arranged along at least a first dimension and a second dimension;
means for performing a hybrid convolution operation that applies a filter across the first block and at least the portion of the second block to generate output data,
wherein the output data includes a first accumulated block and at least a portion of a second accumulated block; and
means for storing the first accumulated block as first output data, wherein the portion of the second block is adjacent to the first block along the first dimension and the portion of the second accumulated block is adjacent to the first accumulated block along the second dimension.

### Claim 29 (depends on 28)

The apparatus of claim 28, wherein the means for retrieving, the means for performing the hybrid convolution operation, and the means for storing are integrated into at least one of a home automation system, a voice-activated device, a wireless speaker and voice activated device, a portable electronic device, a car, a vehicle, a computing device, a communication device, an internet-of-things (IoT) device, a virtual reality (VR) device, a base station, a mobile device, or a combination thereof.

### Claim 30 (depends on 28)

The apparatus of claim 28, wherein the means for performing the hybrid convolution operation is configured to convolve the data with the filter in accordance with at least one of an image processing operation or a neural network convolutional layer operation.