# US 2021/0065393 A1 · Method for stereo matching using end-to-end convolutional neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0065393 A1
- Document ID: US20210065393A1
- Status: Published application
- Kind: A1
- Application number: 17/005,632
- Filed: 2020-08-28
- Published: 2021-03-04
- Earliest claimed: 2019-08-28
- Assignees: Research & Business Foundation Sungkyunkwan University
- Applicants: Research & Business Foundation Sungkyunkwan University
- Inventors: Jaewook JEON; Phuoc Tien NGUYEN; Jinyoung BYUN
- Web page: https://patentgrep.com/patent/US20210065393A1

## CPC

- G06T 7/593: from stereo images
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 9/002: {using neural networks}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06N 3/08: Learning methods
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Disclosed is a stereo matching method for generating a disparity map from a stereo image. The stereo matching method may include obtaining a cost volume by applying a first convolutional neural network (CNN) and a second CNN to a left image and a right image, respectively, wherein the cost volume is determined based on feature maps extracted from the left image and the right image, respectively, performing normalization on the cost volume by applying a third CNN, up-sampling the normalized cost volume, and obtaining a disparity map by applying regression analysis to the up-sampled cost volume.

## Claims

### Claim 1

A stereo matching method for generating a disparity map from a stereo image, comprising:
obtaining a cost volume by applying a first convolutional neural network (CNN) and a second CNN to a left image and a right image, respectively, wherein the cost volume is determined based on feature maps extracted from the left image and the right image, respectively;
performing normalization on the cost volume by applying a third CNN;
up-sampling the normalized cost volume; and
obtaining a disparity map by applying regression analysis to the up-sampled cost volume.

### Claim 2 (depends on 1)

The stereo matching method of claim 1,
wherein the first CNN and the second CNN include an Atrous CNN layer including an extended convolutional layer.

### Claim 3 (depends on 2)

The stereo matching method of claim 2,
wherein the Atrous CNN layer includes a plurality of convolutional layers having different Atrous values, and
wherein the plurality of convolutional layers having the different Atrous values are sequentially applied to a feature map output by a previous CNN layer.

### Claim 4 (depends on 3)

The stereo matching method of claim 3,
wherein the plurality of convolutional layers having the different Atrous values are applied to areas having different sizes based on the Atrous values.

### Claim 5 (depends on 2)

The stereo matching method of claim 2,
wherein the first CNN and the second CNN share a weight applied to each input image.

### Claim 6 (depends on 1)

The stereo matching method of claim 1,
wherein the third CNN is configured as a CNN using a three-dimensional (3D) encoder-decoder.

### Claim 7 (depends on 1)

The stereo matching method of claim 1, further comprising determining depth information of an object included in the stereo image based on the disparity map.

### Claim 8

A stereo matching apparatus comprising:
a processor configured to control the stereo matching apparatus;
a sensor coupled to the processor and configured to obtain a stereo image; and
a memory coupled to the processor and configured to store data,
wherein the processor is configured to:
obtain a cost volume by applying a first convolutional neural network (CNN) and a second CNN to a left image and a right image, respectively, wherein the cost volume is determined based on feature maps extracted from the left image and the right image, respectively, perform normalization on the cost volume by applying a third CNN, up-sample the normalized cost volume, and obtain a disparity map by applying regression analysis to the up-sampled cost volume.

### Claim 9 (depends on 8)

The stereo matching apparatus of claim 8,
wherein the first CNN and the second CNN include an Atrous CNN layer including an extended convolutional layer.

### Claim 10 (depends on 9)

The stereo matching apparatus of claim 9,
wherein the Atrous CNN layer includes a plurality of convolutional layers having different Atrous values, and
wherein the plurality of convolutional layers having the different Atrous values are sequentially applied to a feature map output by a previous CNN layer.

### Claim 11 (depends on 10)

The stereo matching apparatus of claim 10,
wherein the plurality of convolutional layers having the different Atrous values are applied to areas having different sizes based on the Atrous values.

### Claim 12 (depends on 9)

The stereo matching apparatus of claim 9,
wherein the first CNN and the second CNN share a weight applied to each input image.

### Claim 13 (depends on 8)

The stereo matching apparatus of claim 8,
wherein the third CNN is configured as a CNN using a three-dimensional (3D) encoder-decoder.

### Claim 14 (depends on 8)

The stereo matching apparatus of claim 8,
wherein the processor determines depth information of an object included in the stereo image based on the disparity map.
