# US 2024/0257443 A1 · Scene reconstruction from monocular video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0257443 A1
- Document ID: US20240257443A1
- Status: Published application
- Kind: A1
- Application number: 18/524,803
- Filed: 2023-11-30
- Published: 2024-08-01
- Earliest claimed: 2023-01-19
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Christopher B. Choy; Or Litany; Charles Loop; Yuke Zhu; Animashree Anandkumar; Wei Dong
- Web page: https://patentgrep.com/patent/US20240257443A1

## CPC

- G06T 15/20: Perspective computation
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 7/579: from motion
- G06T 7/90: Determination of colour characteristics
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2207/10016: Video; Image sequence
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2210/04: Architectural design, interior design
- G06T 2210/21: Collision detection, intersection
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A technique for reconstructing a three-dimensional scene from monocular video adaptively allocates an explicit sparse-dense voxel grid with dense voxel blocks around surfaces in the scene and sparse voxel blocks further from the surfaces. In contrast to conventional systems, the two-level voxel grid can be efficiently queried and sampled. In an embodiment, the scene surface geometry is represented as a signed distance field (SDF). Representation of the scene surface geometry can be extended to multi-modal data such as semantic labels and color. Because properties stored in the sparse-dense voxel grid structure are differentiable, the scene surface geometry can be optimized via differentiable volume rendering.

## Claims

### Claim 1

A computer-implemented method, comprising:
computing a depth scale for each predicted depth image in a set of predicted depth images corresponding to a set of monocular images comprising a video of a three-dimensional (3D) scene;
calibrating each predicted depth image using the respective depth scale to produce calibrated depth values;
constructing a volumetric grid for the 3D scene storing properties comprising the calibrated depth values and corresponding color values;
rendering the volumetric grid to produce a set of predicted images; and
adjusting the properties to reduce differences between the set of predicted images and the set of monocular images.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising processing the set of monocular images using structure-from-motion supervision to compute the set of predicted depth images.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the properties further comprise a set of predicted normal vector images and a set of semantic images.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein constructing the volumetric grid comprises allocating sparse voxel blocks near surfaces in the 3D scene and storing the properties a dense voxel array within each of the sparse voxel blocks.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprising projecting the sparse voxel blocks to the set of monocular images to associate voxels of the dense voxel arrays with the properties.

### Claim 6 (depends on 4)

The computer-implemented method of claim 4, wherein the sparse voxel blocks are indexed by a collision-free hash map.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein adjusting the volumetric grid comprises updating the calibrated depth values and the corresponding color values according to backpropagated gradients.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the calibrating comprises:
defining a scale function for each predicted depth image in the set of predicted depth images; and
updating the set of predicted depth images to enforce local consistency between visually adjacent monocular images in the set of monocular images.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising, before the rendering, applying a denoising filter to the volumetric grid.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the adjusting of the properties is based on an integral of energy over a surface in the 3D scene that is computed by applying continuous conditional random field smoothing to the calibrated depth values.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the steps of computing, calibrating, constructing, rendering, and adjusting is performed on a server or in a data center to generate the set of predicted images, and at least a portion of the properties stored in the volumetric grid for the 3D scene are streamed to a user device.

### Claim 12 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the steps of computing, calibrating, constructing, rendering, and adjusting is performed within a cloud computing environment.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the steps of computing, calibrating, constructing, rendering, and adjusting is performed for training, testing, or certifying a neural network employed in a machine, robot, or autonomous vehicle.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein at least one of the steps of computing, calibrating, constructing, rendering, and adjusting is performed on a virtual machine comprising a portion of a graphics processing unit.

### Claim 15

A system, comprising:
a memory that stores a set of monocular images comprising a video of a three-dimensional (3D) scene; and
a processor that is connected to the memory, wherein the processor is configured to:
compute a depth scale for each predicted depth image in a set of predicted depth images corresponding to the set of monocular images;
calibrate each predicted depth image using the respective depth scale to produce calibrated depth values;
construct a volumetric grid for the 3D scene storing properties comprising the calibrated depth values and corresponding color values;
render the volumetric grid to produce a set of predicted images; and
adjust the properties to reduce differences between the set of predicted images and the set of monocular images.

### Claim 16 (depends on 15)

The system of claim 15, wherein the properties further comprise a set of predicted normal vector images and a set of semantic images.

### Claim 17 (depends on 15)

The system of claim 15, wherein constructing the volumetric grid comprises allocating sparse voxel blocks near surfaces in the 3D scene and storing the properties a dense voxel array within each of the sparse voxel blocks.

### Claim 18

A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
computing a depth scale for each predicted depth image in a set of predicted depth images corresponding to a set of monocular images comprising a video of a three-dimensional (3D) scene;
calibrating each predicted depth image using the respective depth scale to produce calibrated depth values;
constructing a volumetric grid for the 3D scene storing properties comprising the calibrated depth values and corresponding color values;
rendering the volumetric grid to produce a set of predicted images; and
adjusting the properties to reduce differences between the set of predicted images and the set of monocular images.

### Claim 19 (depends on 18)

The non-transitory computer-readable media of claim 18, wherein the properties further comprise a set of predicted normal vector images and a set of semantic images.

### Claim 20 (depends on 18)

The non-transitory computer-readable media of claim 18, wherein constructing the volumetric grid comprises allocating sparse voxel blocks near surfaces in the 3D scene and storing the properties a dense voxel array within each of the sparse voxel blocks.
