# US 10,310,768 B1 · Convolution calculations in multiple dimensions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,310,768 B1
- Document ID: US10310768B1
- Status: Granted
- Kind: B1
- Application number: 15/403,540
- Filed: 2017-01-11
- Granted: 2019-06-04
- Earliest claimed: 2017-01-11
- Assignees: Ambarella, Inc.
- Applicants: Ambarella, Inc.
- Inventors: Sameer M. Gauria; Peter Verplaetse
- Web page: https://patentgrep.com/patent/US10310768B1

## CPC

- G06F 3/0656: {Data buffering arrangements}
- G06F 3/061: {Improving I/O performance}
- G06F 3/0673: {Single storage device}
- G06F 12/0207: {with multidimensional access, e.g. row/column, matrix}
- G06F 17/153: {Multidimensional correlation or convolution}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 1/60: Memory management
- G06T 7/00: Image analysis
- G06F 2212/1016: Performance improvement
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An apparatus includes a memory and a circuit. The memory may be configured to store data. The circuit generally has a buffer and may be configured to (i) fetch a kernel from the memory, where the kernel may have a plurality of kernel values, (ii) fetch a block from the memory to the buffer, where the block may have a plurality of input tiles and each of the input tiles may have a plurality of input values in multiple dimensions, (iii) calculate a plurality of intermediate values in parallel by multiplying the input tiles read from the buffer with a corresponding one of the kernel values and (iv) calculate an output tile that may have a plurality of output values based on the intermediate values.

## Claims

### Claim 1

An apparatus comprising:
a memory configured to store data;
a first circuit comprising a first buffer and configured to (i) fetch a kernel from said memory to said first buffer, wherein said kernel comprises a plurality of kernel values, and (ii) present said kernel values as stored in said first buffer in a first signal;
a second circuit comprising a second buffer and configured to (i) fetch a block from said memory to said second buffer, wherein said block comprises a plurality of input tiles and each of said input tiles comprises a plurality of input values in multiple dimensions, and (ii) present said input tiles as stored in said second buffer in a second signal, wherein said first circuit fetches said kernel from said memory slower than said second circuits fetches said block from said memory; and
a third circuit configured to (i) receive said kernel values from said first circuit in said first signal, (ii) receive said input tiles from said second circuit in said second signal, (iii) calculate a plurality of intermediate values in parallel by multiplying said input tiles with a corresponding one of said kernel values and (iv) calculate an output tile comprising a plurality of output values based on said intermediate values.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein said second circuit is further configured to (i) read said input tiles repeatedly from said second buffer and (ii) transfer said input tiles repeatedly in said second signal to said third circuit to calculate said output tile.

### Claim 3 (depends on 1)

The apparatus according to claim 1, wherein said output tile comprises (i) a two-dimensional array of output values or (ii) a plurality of two-dimensional arrays of output values arranged in a three-dimensional stack.

### Claim 4 (depends on 1)

The apparatus according to claim 1, wherein said second circuit is further configured to (i) fetch a plurality of additional vectors of said input values from said memory to said second buffer, and (ii) append said additional vectors to said block of said input values in said second buffer to feed calculations of a plurality of additional output tiles.

### Claim 5 (depends on 1)

The apparatus according to claim 1, wherein said third circuit is further configured to reuse said input values received in said second signal from said second buffer across adjacent ones of said input tiles.

### Claim 6 (depends on 1)

The apparatus according to claim 1, wherein said second circuit is further configured to spatially decimate said input tiles prior to transfer in said second signal to said third circuit.

### Claim 7 (depends on 1)

The apparatus according to claim 1, wherein said second circuit is further configured to (i) read from said second buffer at most said input tiles that correspond to one or more non-zero values of said kernel values in said kernel and (ii) transfer in said second signal at most said one or more non-zero values of said kernel values in said kernel.

### Claim 8 (depends on 1)

The apparatus according to claim 1, wherein said block fetched from said memory to said second buffer by said second circuit has at most said input tiles that correspond to one or more non-zero values of said kernel values in said kernel.

### Claim 9 (depends on 1)

The apparatus according to claim 1, wherein said input tiles in said block as stored in said memory are spatially decimated prior to transferring from said memory to said second circuit.

### Claim 10 (depends on 1)

The apparatus according to claim 1, wherein said memory, said first circuit, said second circuit and said third circuit form part of a computer vision system.

### Claim 11

A method for convolution calculations in multiple dimensions, comprising the steps of:
fetching a kernel from a memory to a first buffer in a first circuit, wherein said kernel comprises a plurality of kernel values;
fetching a block from said memory to a second buffer in a second circuit, wherein (i) said block comprises a plurality of input tiles and each of said input tiles comprises a plurality of input values in said multiple dimensions and (ii) said first circuit fetches said kernel from said memory slower than said second circuits fetches said block from said memory;
transferring said kernel values as stored in said first buffer from said first circuit to a third circuit in a first signal;
transferring said input values as stored in said second buffer from said second circuit to said third circuit in a second signal;
calculating a plurality of intermediate values in parallel in said third circuit by multiplying said input tiles with a corresponding one of said kernel values; and
calculating an output tile comprising a plurality of output values in said third circuit based on said intermediate values.

### Claim 12 (depends on 11)

The method according to claim 11, further comprising the steps of:
reading said input tiles repeatedly from said second buffer; and
transferring said input tiles repeatedly in said second signal to said third circuit to calculate said output tile.

### Claim 13 (depends on 11)

The method according to claim 11, wherein said output tile comprises (i) a two-dimensional array of output values or (ii) a plurality of two-dimensional arrays of output values arranged in a three-dimensional stack.

### Claim 14 (depends on 11)

The method according to claim 11, further comprising the steps of:
fetching a plurality of additional vectors of said input values from said memory to said second buffer; and
appending said additional vectors to said block of said input values in said second buffer to feed calculations of a plurality of additional output tiles.

### Claim 15 (depends on 11)

The method according to claim 11, further comprising the step of:
reusing said input values in said third circuit as received in said second signal from said second buffer across adjacent ones of said input tiles.

### Claim 16 (depends on 11)

The method according to claim 11, further comprising the step of:
decimating said input values in said second circuit prior to said transfer in said second signal to said third circuit.

### Claim 17 (depends on 11)

The method according to claim 11, further comprising the steps of:
reading from said second buffer at most said input tiles that correspond to one or more non-zero values of said kernel values in said kernel; and
transferring in said signal at most said input tiles that correspond to said one or more non-zero values of said kernel values in said kernel.

### Claim 18 (depends on 11)

The method according to claim 11, wherein said block fetched from said memory to said second buffer by said second circuit has at most said input tiles that correspond to one or more non-zero values of said kernel values in said kernel.

### Claim 19 (depends on 11)

The method according to claim 11, wherein said input tiles in said block as stored in said memory are decimated prior to transferring from said memory to said second circuit.

### Claim 20 (depends on 11)

The method according to claim 11, wherein said steps are performed in a computer vision system.