# US 2025/0148702 A1 · Method for Monocular Acquisition of Realistic Three-Dimensional Scene Models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0148702 A1
- Document ID: US20250148702A1
- Status: Published application
- Kind: A1
- Application number: 18/937,776
- Filed: 2024-11-05
- Published: 2025-05-08
- Earliest claimed: 2023-11-06
- Assignees: Cinemersive Labs Ltd
- Applicants: Cinemersive Labs Ltd
- Inventors: Victor Lempitskiy; Pavel Solovev
- Web page: https://patentgrep.com/patent/US20250148702A1

## CPC

- G06T 9/001: {Model-based coding, e.g. wire frame}
- G06T 15/40: Hidden part removal
- G06T 9/002: {using neural networks}
- G06T 13/20: Three-dimensional [3D] animation
- G06T 15/04: Texture mapping
- G06T 15/205: {Image-based rendering}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 17/205: {Re-meshing}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Systems and methods for creating a three-dimensional digital model of a scene from a two-dimensional input image of the scene are described. One aspect includes receiving the two-dimensional input image of the scene, and predicting a latent tensor from the input image. The latent tensor comprises three-dimensional geometrical information of the received two-dimensional input image and information about one or more surfaces occluded in the received two-dimensional input image. The latent tensor and the received two-dimensional image may be inputs to the reconstruction neural network that predicts a three-dimensional model. A paired dataset of input images and corresponding latent tensors may be obtained by a joint training of the reconstruction neural network and an encoding neural network that outputs latent tensors from multiview inputs. The process that predicts the latent tensor from the single input image may then be trained on the paired dataset.

## Claims

### Claim 1

A method for creating a three-dimensional digital model of a scene from a two-dimensional input image of the scene, the method comprising:
receiving the two-dimensional input image of the scene;
predicting a latent tensor from the input image, wherein the received two-dimensional input image is input to a prediction process, wherein the latent tensor comprises three-dimensional geometrical information of the received two-dimensional input image and information about one or more surfaces occluded in the received two-dimensional input image; and
reconstructing the three-dimensional digital model via a reconstruction neural network, wherein the latent tensor and the received two-dimensional image are inputs to the reconstruction neural network.

### Claim 2 (depends on 1)

The method of claim 1, wherein the prediction process is a denoising diffusion process, an image-to-image translation network, or an auto-regressive network.

### Claim 3 (depends on 1)

The method of claim 1, wherein the reconstruction neural network is a deep neural network comprised of one or more convolutional layers, one or more self-attention layers, or one or more cross-attention layers.

### Claim 4 (depends on 1)

The method of claim 1, further comprising generating a paired training data set, the generating further comprising:
obtaining a plurality of auxiliary images of a training scene taken from different viewpoints;
inputting each of the plurality of auxiliary images and a reference image into an encoder neural network; and
obtaining, via an output of the encoder neural network, a trained latent tensor that comprises three-dimensional geometrical information about the training scene and information about one or more surfaces occluded in the reference image, wherein the reference image and the trained latent tensor constitute an entry in the paired training data set.

### Claim 5 (depends on 4)

The method of claim 4, further comprising joint training of the encoder neural network and the reconstruction neural network, the method comprising:
reconstructing a three-dimensional digital training model via the reconstruction neural network, wherein the paired training data set entries are inputs to the reconstruction neural network;
rendering a two-dimensional auxiliary image via a differentiable renderer wherein the three-dimensional digital training model is input into the differentiable renderer;
calculating a loss function via a comparison of the rendered two-dimensional auxiliary image and at least one of the plurality of auxiliary images of the training scene or the reference image; and
inputting the calculated loss function into any combination of the encoder neural network, prediction process, and the reconstruction neural network.

### Claim 6 (depends on 4)

The method of claim 4, wherein the prediction process is trained on the paired training dataset, such that a corresponding trained latent tensor is computed for each of the plurality of auxiliary images.

### Claim 7

A system for creating a three-dimensional digital model of a scene from a two-dimensional input image of the scene, the system comprising:
a processor configured to execute a prediction process to predict a latent tensor based on the two-dimensional input image, wherein the latent tensor comprises three-dimensional geometrical information of the received two-dimensional input image and information about one or more surfaces occluded in the received two-dimensional input image; and
a reconstruction neural network configured to receive the latent tensor and the received two-dimensional image as inputs, wherein reconstruction neural network is further configured to reconstruct the three-dimensional digital model.

### Claim 8 (depends on 7)

The system of claim 7, wherein the prediction process is implemented via any of denoising diffusion process, an image translation network, or an auto-regressive network.

### Claim 9 (depends on 7)

The system of claim 7, wherein the reconstruction neural network is a deep neural network comprised of one or more convolutional layers, one or more self-attention layers, or one or more cross-attention layers.

### Claim 10 (depends on 7)

The system of claim 7, wherein the system further comprises:
an encoder network configured to receive a plurality of auxiliary images of a training scene taken from different viewpoints and a reference image as input, said encoder network further configured to output a trained latent tensor that comprises three-dimensional geometrical information about the training scene and the information about one or more surfaces occluded in the reference image, wherein the reference image and the trained latent tensor constitute an entry in the paired training data set.

### Claim 11 (depends on 10)

The system of claim 10, further comprising:
the reconstruction network configured to receive the trained data set and output a reconstructed three-dimensional training model of the training scene;
a differentiable renderer configured to receive as input the three-dimensional digital training model and render a two-dimensional auxiliary image based on said three-dimensional digital training model;
a processor or a loss function unit configured to calculate a loss function based on a comparison of the rendered two-dimensional auxiliary image of the training scene and at least one of the plurality of auxiliary images or the reference image; and
any combination of the encoder neural network, prediction process and the reconstruction neural network further configured to receive the calculated loss function for training.

### Claim 12 (depends on 1)

A machine-readable storage medium storing a set of instructions that are executable by one or more processors of a system for creating a three-dimensional digital model of a scene from a two-dimensional input image of the scene, wherein the set of instructions are configured to perform the method of claim 1.

### Claim 13

A method for training a prediction process, the method comprising:
obtaining a plurality of auxiliary images of a training scene taken from different viewpoints;
inputting each of the plurality of auxiliary images and a reference image into an encoder neural network;
obtaining, via an output of the encoder neural network, a trained latent tensor that comprises three-dimensional geometrical information about the training scene and information about one or more surfaces occluded in the reference image, wherein the reference image and the latent tensor constitute an entry in the paired training data set.

### Claim 14 (depends on 13)

The method of claim 13, further comprising jointly training the encoder neural network and the reconstruction neural network, the method comprising:
reconstructing a three-dimensional digital training model via the reconstruction neural network, wherein the paired training data set entries are inputs to the reconstruction neural network;
rendering a two-dimensional auxiliary image via a differentiable renderer, wherein the three-dimensional digital training model is input into the differentiable renderer;
calculating a loss function via a comparison of the rendered two-dimensional auxiliary image and at least one of the plurality of auxiliary images of the scene or the reference image; and
inputting the calculated loss function into any combination of the encoder neural network, prediction process and the reconstruction neural network.

### Claim 15

A system for training a prediction process, the method comprising:
an encoder network configured to receive a plurality of auxiliary images of a training scene taken from different viewpoints and a reference image as input, said encoder network further configured to output a trained latent tensor that comprises three-dimensional geometrical information about the training scene and information about one or more surfaces occluded in the reference image, wherein the reference image and the trained latent tensor constitute an entry in the paired training data set, and the prediction process is trained via the paired training data set.

### Claim 16 (depends on 15)

The system of claim 15 further being configured to jointly train the encoder neural network and a reconstruction neural network, the system further comprising:
a reconstruction network configured to receive the trained data set and output a reconstructed three-dimensional training model of the training scene;
a differentiable renderer configured to receive as input the three-dimensional digital training model and render a two-dimensional auxiliary image based on said three-dimensional digital training model;
a processor or a loss function unit configured to calculate a loss function based on a comparison of the rendered two-dimensional auxiliary image of the training scene and at least one of the plurality of auxiliary images or the reference image; and
any combination of the encoder neural network, prediction process and the reconstruction neural network further configured to receive the calculated loss function for training.

### Claim 17 (depends on 13)

A machine-readable storage medium storing a set of instructions that are executable by one or more processors of a system for training, wherein the set of instructions are configured to perform the method of claim 13.
