# US 2021/0279943 A1 · Systems and methods for end to end scene reconstruction from multiview images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279943 A1
- Document ID: US20210279943A1
- Status: Published application
- Kind: A1
- Application number: 17/193,822
- Filed: 2021-03-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-05
- Assignees: Magic Leap, Inc.
- Applicants: Magic Leap, Inc.
- Inventors: Zachary Paul MUREZ
- Web page: https://patentgrep.com/patent/US20210279943A1

## CPC

- G06T 15/08: Volume rendering
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06N 3/02: Neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 7/55: from multiple images
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10024: Color image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods of generating a three-dimensional (3D) reconstruction of a scene or environment surrounding a user of a spatial computing system, such as a virtual reality, augmented reality or mixed reality system, using only multiview images comprising RGB images, and without the need for depth sensors or depth data from sensors. Features are extracted from a sequence of frames of RGB images and back-projected using known camera intrinsics and extrinsics into a 3D voxel volume wherein each pixel of the voxel volume is mapped to a ray in the voxel volume. The back-projected features are fused into the 3D voxel volume. The 3D voxel volume is passed through a 3D convolutional neural network to refine the and regress truncated signed distance function values at each voxel of the 3D voxel volume.

## Claims

### Claim 1

A method of generating a three-dimensional (3D) reconstruction of a scene from multiview images, the method comprising:
obtaining a sequence of frames of red green blue (RGB) images;
extracting features from the sequence of frames of RGB images using a two-dimensional convolutional neural network (2D CNN);
back-projecting the features from each frame using known camera intrinsics and extrinsics into a 3D voxel volume wherein each pixel of the voxel volume is mapped to a ray in the voxel volume;
fusing/accumulating features from each frame into the 3D voxel volume;
passing the 3D voxel volume through a 3D convolutional neural network (3D CNN) having an encoder-decoder to refine the features in the 3D voxel volume and regress output truncated signed distance function (TSDF) values at each voxel of the 3D voxel volume.

### Claim 2 (depends on 1)

The method of claim 1, wherein the frames are fused into a single 3D feature volume using a running average.

### Claim 3 (depends on 2)

The method of claim 2, wherein the running average is a simple running average.

### Claim 4 (depends on 3)

The method of claim 3, wherein the running average is a weighted running average.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
after passing the 3D voxel volume through all layers of the 3D CNN, passing the refined features in the 3D voxel volume and TSDF values at each voxel of the 3D voxel volume through a batch normalization (batchnorm) function and a rectified linear unit (reLU) function.

### Claim 6 (depends on 1)

The method of claim 1, wherein additive skip connections are included from an encoder to a decoder of the 3D CNN, and the method further comprises:
using the additive skip connections to skip one or more features in the 3D voxel volume from the encoder to the decoder of the 3D CNN.

### Claim 7 (depends on 6)

The method of claim 6, wherein one or more null voxels of the 3D voxel volume do not have features back-projected into them corresponding to voxels which were not observed during the sequence of frames of RGB images, and the method further comprises:
not using the additive skip connections from the encoder for the null voxels additive skip connection for the null voxels;
passing the null voxels through the batchnorm function and relu function to match the magnitude of the voxels undergoing the skip connections.

### Claim 8 (depends on 1)

The method of claim 1, wherein the 3D CNN has a plurality of layers each having a set of 3×3×3 residual blocks, and the 3D CNN implements downsampling with 3×3×3 stride 2 convolution and upsampling using trilinear interpolation followed by a 1×1×1 convolution.

### Claim 9 (depends on 1)

The method of claim 1, wherein the 3D CNN further comprises an additional head for predicting semantic segmentation, and the method further comprises:
the 3D CNN predicting semantic segmentation of the features in the 3D voxel volume.

### Claim 10 (depends on 1)

The method of claim 1, further comprising training the 2D CNN using short frame sequences covering portions of scenes.

### Claim 11 (depends on 10)

The method of claim 10, wherein the short frame sequences include ten or fewer frame sequences.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
fine tuning the training of the 2D CNN using larger frame sequences having more frame sequences that the short frame sequences.

### Claim 13 (depends on 12)

The method of claim 12, wherein the larger frame sequences include 100 or more frame sequences.

### Claim 14

A cross reality system, comprising:
a head-mounted display device having a display system;
a computing system in operable communication with the head-mounted display;
a plurality of camera sensors in operable communication with the computing system;
wherein the computing system is configured to generate a three-dimensional (3D) reconstruction of the scene from a sequence of frames of RGB images captured by the camera sensors by a process comprising:
obtaining a sequence of a frames of red green blue (RGB) images of a scene within a field of view of the camera sensors from the camera sensors;
extracting features from the sequence of frames of RGB images using a two-dimensional convolutional neural network (2D CNN);
back-projecting the features from each frame using known camera intrinsics and extrinsics into a 3D voxel volume wherein each pixel of the voxel volume is mapped to a ray in the voxel volume;
fusing the features from each frame into the 3D voxel volume;
passing the 3D voxel volume through a 3D convolutional neural network (3D CNN) having an encoder-decoder to refine the features in the 3D voxel volume and regress output truncated signed distance function (TSDF) values at each voxel of the 3D voxel volume.

### Claim 15 (depends on 14)

The system of claim 14, wherein the frames are fused into a single 3D feature volume using a running average.

### Claim 16 (depends on 15)

The system of claim 15, wherein the running average is a simple running average.

### Claim 17 (depends on 16)

The system of claim 16, wherein the running average is a weighted running average.

### Claim 18 (depends on 14)

The system claim 14, wherein the process for generating a three-dimensional (3D) reconstruction of the scene from the sequence of frames of RGB images further comprises:
after passing the 3D voxel volume through all layers of the 3D convolutional encoder-decoder, passing the refined features in the 3D voxel volume and TSDF values at each voxel of the 3D voxel volume through a batch normalization (batchnorm) function and a rectified linear unit (reLU) function.

### Claim 19 (depends on 14)

The system of claim 14, wherein additive skip connections are included from an encoder to a decoder of the 3D CNN, and the process for generating a three-dimensional (3D) reconstruction of the scene from the sequence of frames of RGB images further comprises:
using the additive skip connections to skip one or more features in the 3D voxel volume from the encoder to the decoder of the 3D CNN.

### Claim 20 (depends on 19)

The system of claim 19, wherein one or more null voxels of the 3D voxel volume do not have features back-projected into them corresponding to voxels which were not observed during the sequence of frames of RGB images, and the process for generating a three-dimensional (3D) reconstruction of the scene from the sequence of frames of RGB images further comprises:
not using the additive skip connections from the encoder for the null voxels additive skip connection for the null voxels;
passing the null voxels through the batchnorm function and reLU function to match the magnitude of the voxels undergoing the skip connections.

### Claim 21 (depends on 14)

The system of claim 14, wherein the 3D CNN further comprises an additional head for predicting semantic segmentation, and the method further comprises:
the 3D CNN predicting semantic segmentation of the features in the 3D voxel volume.