# US 2021/0279904 A1 · Systems and methods for depth estimation by learning triangulation and densification of sparse points for multi-view stereo

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279904 A1
- Document ID: US20210279904A1
- Status: Published application
- Kind: A1
- Application number: 17/194,117
- Filed: 2021-03-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-05
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Ayan Tuhinendu SINHA
- Web page: https://patentgrep.com/patent/US20210279904A1

## CPC

- G06T 7/596: {from three or more stereo images}
- H04N 13/128: Adjusting depth or disparity
- G06T 7/593: from stereo images
- G06T 7/73: using feature-based methods
- G06T 9/002: {using neural networks}
- H04N 13/15: for colour aspects of image signals
- H04N 13/161: Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597)
- H04N 13/282: for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
- G06T 2207/10012: Stereo images
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- H04N 2013/0081: {Depth or disparity estimation from stereoscopic image signals}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Systems and methods for estimating depths of features in a scene or environment surrounding a user of a spatial computing system, such as a virtual reality, augmented reality or mixed reality (collectively, cross reality) system, in an end-to-end process. The estimated depths can be utilized by a spatial computing system, for example, to provide an accurate and effective 3D cross reality experience.

## Claims

### Claim 1

A method for estimating depth of features in a scene from multi-view images, the method comprising:
obtaining multi-view images, including an anchor image of the scene and a set of reference images of the scene;
passing the anchor image and reference images through a shared RGB encoder and descriptor decoder which (1) outputs a respective descriptor field of descriptors for the anchor image and each reference image, (ii) detects interest points in the anchor image in conjunction with relative poses to determine a search space in the reference images from alternate view-points, and (iii) outputs intermediate feature maps;
sampling the respective descriptors in the search space of each reference image to determine descriptors in the search space and matching the identified descriptors with descriptors for the interest points in the anchor image, such matched descriptors referred to as matched keypoints;
triangulating the matched keypoints using singular value decomposition (SVD) to output 3D points;
passing the 3D points through a sparse depth encoder to create a sparse depth image from the 3D points and output feature maps; and
a depth decoder generating a dense depth image based on the output feature maps for the sparse depth encoder and the intermediate feature maps from the RGB encoder.

### Claim 2 (depends on 1)

The method of claim 1, wherein the shared RGB encoder and descriptor decoder comprises two encoders including an RGB image encoder and a sparse depth image encoder, and three decoders including an interest point detection encoder, a descriptor decoder, and a dense depth prediction encoder.

### Claim 3 (depends on 1)

The method of claim 1, wherein the shared RGB encoder and descriptor decoder is a fully-convolutional neural network configured to operate on a full resolution of the anchor image and transaction images.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
feeding the feature maps from the RGB encoder into a first task-specific decoder head to determine weights for the detecting of interest points in the anchor image and outputting interest point descriptions.

### Claim 5 (depends on 1)

The method of claim 1, wherein the descriptor decoder comprises a U-Net like architecture to fuse fine and course level image information for matching the identified descriptors with descriptors for the interest points.

### Claim 6 (depends on 1)

The method of claim 1, wherein the search space is constrained to a respective epipolar line in the reference images plus a fixed offset on either side of the epipolar line, and within a feasible depth sensing range along the epipolar line.

### Claim 7 (depends on 1)

The method of claim 1, wherein bilinear sampling is used by the shared RGB encoder and descriptor decoder to output the respective descriptors at desired points in the descriptor field.

### Claim 8 (depends on 1)

The method of claim 1, wherein the step of triangulating the matched keypoints comprises:
estimating respective two dimensional (2D) positions of the interest points by computing a softmax across spatial axes to output cross-correlation maps;
performing a soft-argmax operation to calculate the 2D position of joints as a center of mass of corresponding cross-correlation maps;
performing a linear algebraic triangulation from the 2D estimates; and
using a singular value decomposition (SVD) to output 3D points.

### Claim 9

A cross reality system, comprising:
a head-mounted display device having a display system;
a computing system in operable communication with the head-mounted display;
a plurality of camera sensors in operable communication with the computing system;
wherein the computing system is configured to estimate depths of features in a scene from a plurality of multi-view images captured by the camera sensors by a process comprising:
obtaining a multi-view images, including an anchor image of the scene and a set of reference images of a scene within a field of view of the camera sensors from the camera sensors;
passing the anchor image and reference images through a shared RGB encoder and descriptor decoder which (1) outputs a respective descriptor field of descriptors for the anchor image and each reference image, (ii) detects interest points in the anchor image in conjunction with relative poses to determine a search space in the reference images from alternate view-points, and (iii) outputs intermediate feature maps;
sampling the respective descriptors in the search space of each reference image to determine descriptors in the search space and matching the identified descriptors with descriptors for the interest points in the anchor image, such matched descriptors referred to as matched keypoints;
triangulating the matched keypoints using singular value decomposition (SVD) to output 3D points;
passing the 3D points through a sparse depth encoder to create a sparse depth image from the 3D points and output feature maps; and
a depth decoder generating a dense depth image based on the output feature maps for the sparse depth encoder and the intermediate feature maps from the RGB encoder.

### Claim 10 (depends on 9)

The cross reality system of claim 9, wherein the shared RGB encoder and descriptor decoder comprises two encoders including an RGB image encoder and a sparse depth image encoder, and three decoders including an interest point detection encoder, a descriptor decoder, and a dense depth prediction encoder.

### Claim 11 (depends on 9)

The cross reality system of claim 9, wherein the shared RGB encoder and descriptor decoder is a fully-convolutional neural network configured to operate on a full resolution of the anchor image and transaction images.

### Claim 12 (depends on 9)

The cross reality system of claim 9, wherein the process for estimating depths of features in a scene from a plurality of multi-view images captured by the camera sensors further comprises:
feeding the feature maps from the RGB encoder into a first task-specific decoder head to determine weights for the detecting of interest points in the anchor image and outputting interest point descriptions.

### Claim 13 (depends on 9)

The cross reality system of claim 9, wherein the descriptor decoder comprises a U-Net like architecture to fuse fine and course level image information for matching the identified descriptors with descriptors for the interest points.

### Claim 14 (depends on 9)

The cross reality system of claim 9, wherein the search space is constrained to a respective epipolar line in the reference images plus a fixed offset on either side of the epipolar line, and within a feasible depth sensing range along the epipolar line.

### Claim 15 (depends on 9)

The cross reality system of claim 9, wherein bilinear sampling is used by the shared RGB encoder and descriptor decoder to output the respective descriptors at desired points in the descriptor field.

### Claim 16 (depends on 9)

The cross reality system of claim 9, wherein the step of triangulating the matched keypoints comprises:
estimating respective two dimensional (2D) positions of the interest points by computing a softmax across spatial axes to output cross-correlation maps;
performing a soft-argmax operation to calculate the 2D position of joints as a center of mass of corresponding cross-correlation maps;
performing a linear algebraic triangulation from the 2D estimates; and
using a singular value decomposition (SVD) to output 3D points.
