# US 12,002,229 B2 · Accelerating speckle image block matching using convolution techniques

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,229 B2
- Document ID: US12002229B2
- Status: Granted
- Kind: B2
- Application number: 17/479,034
- Filed: 2021-09-20
- Published: 2023-03-09
- Granted: 2024-06-04
- Earliest claimed: 2021-09-09
- Assignees: Ambarella International LP
- Applicants: Ambarella International LP
- Inventors: Wenhai Gao; Liangliang Wang; Bo Yu
- Web page: https://patentgrep.com/patent/US12002229B2

## CPC

- G06T 7/521: from laser ranging, e.g. using interferometry; from the projection of structured light
- G06N 3/02: Neural networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06V 10/443: {by matching or filtering}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/50: by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
- G06V 10/82: using neural networks
- G06T 2207/10024: Color image
- G06T 2207/10048: Infrared image
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus comprising an interface, a light projector and a processor. The interface may be configured to receive pixel data. The light projector may be configured to generate a structured light pattern. The processor may be configured to process the pixel data arranged as video frames and generate disparity and depth maps. The processor may comprise convolutional neural network hardware that may arrange reference images into a tensor, perform logical operations on one of the video frames in a depth direction of the tensor to generate a tensor of feature maps of the video frames, use a convolution to reduce an amount of calculations performed in the depth direction of the tensor of feature maps, perform convolution filtering on the tensor of the feature maps, determine an index map location, and search lookup data based on the index map location to determine the disparity and depth maps.

## Claims

### Claim 1

An apparatus comprising:
an interface configured to receive pixel data;
a structured light projector configured to generate a structured light pattern; and
a processor configured to (i) process said pixel data arranged as video frames, (ii) preprocess a plurality of reference images comprising said structured light pattern, (iii) generate lookup data and (iv) perform convolution operations to generate a disparity map and a depth map in response to (a) said video frames, (b) said structured light pattern, (c) said reference images and (d) said lookup data, wherein said processor comprises convolutional neural network hardware configured to:
(A) arrange said plurality of reference images into a four-dimensional tensor,
(B) perform logical operations on one of said video frames in a depth direction of said four-dimensional tensor to generate a tensor of feature maps of said video frames,
(C) use a horizontal and vertical separation convolution and dilation to reduce an amount of calculations performed to implement convolution filtering in said depth direction of said tensor of feature maps,
(D) perform said convolution filtering on said tensor of said feature maps,
(E) determine a depth dimension index value in response to a maximum value calculation performed in said depth direction of said tensor of feature maps,
(F) determine an index map location, and
(G) search said lookup data based on said index map location to determine said disparity map and said depth map.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein said four-dimensional tensor comprises (i) a depth dimension representing a depth of offset images, (ii) a first dimension representing a total number of said reference images, (iii) a second dimension representing a height of each of said reference images and (iv) a third dimension representing a width of each of said reference images.

### Claim 3 (depends on 2)

The apparatus according to claim 2, wherein said tensor of said feature maps is generated having a same size as said four-dimensional tensor.

### Claim 4 (depends on 1)

The apparatus according to claim 1, wherein arranging said reference images into said four-dimensional tensor enables said logical operations to be performed in parallel in order to generate said tensor of feature maps.

### Claim 5 (depends on 1)

The apparatus according to claim 1, wherein said horizontal and vertical separation convolution and said dilation are implemented to reduce an amount of calculations performed by said convolutional neural network hardware in response to (i) separating a two-dimensional convolution kernel into a cascade of an x-axis one-dimensional convolution kernel and a y-axis one-dimensional convolution kernel, (ii) decomposing said x-axis one-dimensional convolution kernel into a first one-dimensional convolution kernel and a first one-dimensional hollow kernel and (iii) decomposing said y-axis one-dimensional convolution kernel into a second one-dimensional convolution kernel and a second one-dimensional hollow kernel.

### Claim 6 (depends on 5)

The apparatus according to claim 5, wherein said convolutional neural network hardware is further configured to calculate a cost aggregation value in a direction in response to said convolution operations of one or more of said first one-dimensional convolution kernel, said first one-dimensional hollow kernel, said second one-dimensional convolution kernel and said second one-dimensional hollow kernel.

### Claim 7 (depends on 6)

The apparatus according to claim 6, wherein (i) said cost aggregation value comprises a cost four-dimensional tensor having a same size as said four-dimensional tensor and (ii) an image boundary of said video frames is filled.

### Claim 8 (depends on 5)

The apparatus according to claim 5, wherein said first one-dimensional convolution kernel, said first one-dimensional hollow kernel, said second one-dimensional convolution kernel and said second one-dimensional hollow kernel are each a smaller size than said two-dimensional convolution kernel.

### Claim 9 (depends on 8)

The apparatus according to claim 8, wherein said amount of calculations performed by said convolution filtering is achieved by reducing a number of repetitive calculations performed in said convolution operations.

### Claim 10 (depends on 1)

The apparatus according to claim 1, wherein (i) said lookup data comprises a disparity table and a depth table, (ii) said disparity table and said depth table are each one-dimensional tables and (iii) said disparity table and said depth table each have a size of 1×D, where D is a depth dimension of said four-dimensional tensor.

### Claim 11 (depends on 1)

The apparatus according to claim 1, wherein (i) said lookup data comprises a disparity table and a depth table and (ii) said convolutional neural network hardware is configured to search said disparity table and said depth table separately.

### Claim 12 (depends on 1)

The apparatus according to claim 1, wherein said four-dimensional tensor of said reference images and said lookup data are prepared offline before said video frames are received.

### Claim 13 (depends on 12)

The apparatus according to claim 12, wherein (i) said reference images are shifted pixel-by-pixel in an X direction and a Y direction according to prior conditions of said structured light pattern to generate offset images and (ii) said offset images are arranged in an order into said four-dimensional tensor of reference images.

### Claim 14 (depends on 13)

The apparatus according to claim 13, wherein said order of said offset images is from a minimum disparity to a maximum disparity.

### Claim 15 (depends on 1)

The apparatus according to claim 1, wherein a maximum depth dimension index map in said depth direction of said tensor of said feature maps is configured to quickly pass through said lookup data to enable said processor to obtain said disparity map and said depth map.

### Claim 16 (depends on 1)

The apparatus according to claim 1, wherein said logical operations comprise performing an XOR operation between said reference images in said four-dimensional tensor and said video frames.

### Claim 17 (depends on 1)

The apparatus according to claim 1, wherein said processor is configured to perform preprocessing on said video frames before said convolutional neural network hardware performs said logical operations.

### Claim 18 (depends on 1)

The apparatus according to claim 1, wherein (i) said reference images in said four-dimensional tensor are used to perform pattern matching with said video frames to determine an offset and (ii) said offset is used to search said lookup data.

### Claim 19 (depends on 1)

The apparatus according to claim 1, wherein (i) said depth dimension index value comprises a maximum value position for every pixel in one of said video frames, (ii) said depth dimension index value is arranged in a tensor of width w and height h and to determine said index map location and (iii) said index map location is used to search said lookup data to find a disparity and depth of said every pixel.

### Claim 20 (depends on 1)

The apparatus according to claim 1, wherein operations performed by said convolutional neural network hardware is configured to provide hardware acceleration for speckle image block matching using convolution techniques.