# US 2021/0174524 A1 · Systems and methods for depth estimation via affinity learned with convolutional spatial propagation networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174524 A1
- Document ID: US20210174524A1
- Status: Published application
- Kind: A1
- Application number: 16/770,612
- Filed: 2018-06-29
- Published: 2021-06-10
- Earliest claimed: 2018-06-29
- Assignees: Baidu USA LLC; Baidu.Com Times Technology (Beijing) Co., Ltd.
- Applicants: Baidu USA, LLC; Baidu.com Times Technology (Beijing) Co., Ltd.
- Inventors: Peng WANG; Xinjing CHENG; Ruigang YANG
- Web page: https://patentgrep.com/patent/US20210174524A1

## CPC

- G06T 7/50: Depth or shape recovery
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Presented are systems and methods for improving speed and quality of real-time per-pixel depth estimation of scene layouts from a single image by using an end-to-end Convolutional Spatial Propagation Network (CSPN). An efficient linear propagation model performs propagation using a recurrent convolutional operation. The affinity among neighboring pixels may be learned through a deep convolutional neural network (CNN). The CSPN may be applied to two depth estimation tasks, given a single image: (1) to refine the depth output of existing methods, and (2) to convert sparse depth samples to a dense depth map, e.g., by embedding the depth samples within the propagation procedure. The conversion ensures that the sparse input depth values are preserved in the final depth map and runs in real-time and is, thus, well suited for robotics and autonomous driving applications, where sparse but accurate depth measurements, e.g., from LiDAR, can be fused with image data.

## Claims

### Claim 1

A method for end-to-end training a model for estimating depth from a single image, the method comprising:
receiving, at a convolutional spatial propagation network (CSPN), an affinity matrix and an initial depth map that comprises pixel depths associated with structures in an image;
using, by the CSPN, the affinity matrix and the initial depth map to propagate a local area in a plurality of directions while updating, in parallel, each pixel depth recurrently to improve, for a set of boundaries in a refined depth map, an alignment with a set of boundaries of the structures in the image;
outputting, at a final layer of the CSPN, the refined depth map;
calculating error;
performing backpropagation; and
iterating until a stop condition is reached.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising inputting the image into a first convolutional neural network (CNN) to output the initial depth map.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the propagation operation learns a diffusion tensor through the CSPN.

### Claim 4 (depends on 3)

The method according to claim 3, wherein the CSPN uses a propagation model that is an anisotropic diffusion process and uses a recurrent convolutional operation to obtain a long-range context.

### Claim 5 (depends on 2)

The method according to claim 2, further comprising adding mirror connections to the first CNN by direct concatenating a feature from an encoder to up-projection layers.

### Claim 6 (depends on 3)

The method according to claim 3, further comprising inputting into the first CNN sparse depth values that correspond to ground truth depth values associated with sparse set of pixels and combining the sparse depth values with the image data to output the initial depth map.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the CSPN generates the refined depth map in response to receiving the sparse depth values.

### Claim 8 (depends on 6)

The method according to claim 6, wherein the convolution by the recurrent network reduces depth gradient errors of surrounding pixels thereby reducing the likelihood of discontinuities when replacing values in the refined depth map with sparse depth values.

### Claim 9 (depends on 6)

The method according to claim 6, wherein the ground truth depth values have been obtained through measurements by one of a light detection and ranging system and a line laser senor system.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the affinity matrix is generated by a second CNN that receives the image and that comprises parameters that are learned.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the diffusion process comprises learning a diffusion tensor through a deep CNN from the image data.

### Claim 12

A method for estimating depth from a single image, the method comprising:
inputting an image into a model that has been trained end-to-end by using a convolutional spatial propagation network (CSPN) that converts an initial depth map that comprises pixel depths associated with structures in an image into a refined depth map by propagating a local area in a plurality of directions while recurrently updating, in parallel, each pixel depth; and
using the refined depth map to estimate pixel depth from the image.

### Claim 13 (depends on 12)

The method according to claim 12, wherein the CSPN receives sparse depth samples and uses the sparse depth samples to generate the refined depth map.

### Claim 14 (depends on 13)

The method according to claim 13, wherein values of the sparse depth samples are preserved in the refined depth map.

### Claim 15 (depends on 12)

The method according to claim 12, wherein the estimated pixel depth represents a distance between a structure and a camera.

### Claim 16

A Convolutional Spatial Propagation Network (CSPN) system comprising:
a processor; and
a non-transitory computer-readable medium comprising instructions that, when executed by the processor, cause steps to be performed, the steps comprising:
receiving, by a convolutional neural network (CNN), an image to generate an initial depth map;
receiving, by an input layer, at least an affinity matrix and the initial depth map, the initial depth map comprising pixel depths associated with structures in the image;
using a recurrent neural network (RNN) to propagate a local area in a plurality of directions while updating, in parallel, each pixel depth to improve, for a set of boundaries in a refined depth map, an alignment with a set of boundaries of the structures in the image; and
outputting, at a final layer, refined depth map.

### Claim 17 (depends on 16)

The system according to claim 16, further comprising one or more depth sensors that measure depth samples, the CSPN uses the sparse depth samples to generate the refined depth map.

### Claim 18 (depends on 17)

The system according to claim 17, wherein values of the sparse depth samples are preserved in the refined depth map.

### Claim 19 (depends on 16)

The system according to claim 16, wherein the two or more of the pixel depths are updated using the propagation operation.

### Claim 20 (depends on 19)

The system according to claim 19, wherein the propagation process is an anisotropic diffusion process that propagates a local area in a plurality of directions.
