# US 2025/0061730 A1 · Scene reconstruction in three-dimensions from two-dimensional images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0061730 A1
- Document ID: US20250061730A1
- Status: Published application
- Kind: A1
- Application number: 18/936,477
- Filed: 2024-11-04
- Published: 2025-02-20
- Earliest claimed: 2019-06-17
- Assignees: Snap Inc.
- Applicants: Snap Inc.
- Inventors: Riza Alp Guler; Georgios Papandreou; Iason Kokkinos
- Web page: https://patentgrep.com/patent/US20250061730A1

## CPC

- G06T 7/50: Depth or shape recovery
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06T 7/73: using feature-based methods
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06V 2201/07: Target detection
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

This specification relates to reconstructing three-dimensional (3D) scenes from two-dimensional (2D) images using a neural network. According to a first aspect of this specification, there is described a method for creating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, the method comprising: receiving a single two-dimensional image; identifying all objects in the image to be reconstructed and identifying the type of said objects; estimating a three-dimensional representation of each identified object; estimating a three-dimensional plane physically supporting all three-dimensional objects; and positioning all three-dimensional objects in space relative to the supporting plane.

## Claims

### Claim 1

A method comprising:
extracting object/person re-identification (REID) embeddings from object/human crops by a teacher-student network;
using the REID embeddings as a supervision signal to train a REID branch for a neural network; and
processing the REID embeddings generated by the teacher-student network to track object/person identities across multiple images in a scene having object overlaps and occlusions.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
creating a three-dimensional reconstruction of the scene with multiple objects from a single two-dimensional image;
estimating a three-dimensional representation of each identified object using a deep machine learning model;
estimating a three-dimensional plane physically supporting the multiple objects based on three-dimensional positions of the three-dimensional representation;
measuring an error value based on comparing the three-dimensional plane with a two-dimensional plane shown in the single two-dimensional image; and
adjusting the three-dimensional plane based on the error value.

### Claim 3 (depends on 2)

The method of claim 2, wherein the deep machine learning model includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.

### Claim 4 (depends on 3)

The method of claim 3, wherein the deep machine learning model predicts three-dimensional landmark positions of multiple objects by concatenating feature data from one or more intermediate layers of a neural network and the predicted three-dimensional landmark positions are estimated simultaneously for a predicted type of object depicted in each region.

### Claim 5 (depends on 2)

The method of claim 2, wherein the estimating the three-dimensional plane supporting the multiple objects is performed for a sequence of frames using relative camera pose estimation and plane localization using correspondences between points of consecutive frames.

### Claim 6 (depends on 2)

The method of claim 2, further comprising receiving a plurality of images, wherein the estimating the three-dimensional representations of the multiple objects and displaying the three-dimensional representations relative to the three-dimensional plane are done for each received image in real-time.

### Claim 7 (depends on 6)

The method of claim 6, wherein the deep machine learning model comprises one or more hidden layers, and the method further comprises combining hidden layer responses at consecutive frames by averaging the hidden layer responses.

### Claim 8 (depends on 2)

The method of claim 2, wherein digital graphics objects are synthetically added to the three-dimensional reconstruction of the scene, in a given relation to the three-dimensional positions, and then projected back to the single two-dimensional image.

### Claim 9 (depends on 1)

The method of claim 1, wherein the teacher-student network is trained using a pre-trained REID network to generate high-dimensional embeddings.

### Claim 10 (depends on 1)

The method of claim 1, further comprising using the REID embeddings to maintain object/person identity across a sequence of video frames.

### Claim 11 (depends on 1)

The method of claim 1, wherein the REID branch of the neural network is fully convolutional, allowing for real-time processing independent of a number of objects in the scene.

### Claim 12 (depends on 1)

The method of claim 1, further comprising integrating the REID embeddings with a tracking algorithm to handle occlusions and reappearances of objects/persons in video frames.

### Claim 13 (depends on 1)

The method of claim 1, wherein the REID embeddings are used to associate virtual objects or skins with tracked objects/persons across video frames.

### Claim 14 (depends on 1)

The method of claim 1, further comprising using the REID embeddings to enhance accuracy of object/person-specific parameter smoothing over time.

### Claim 15 (depends on 1)

The method of claim 1, wherein the REID embeddings are used to generate a discriminative signature for each object/person, invariant to changes in camera position and lighting conditions.

### Claim 16 (depends on 1)

The method of claim 1, further comprising employing a memory-based online learning approach to update the REID embeddings as new video frames are processed.

### Claim 17

A system comprising:
a memory; and
at least one processor, wherein the at least one processor is configured to perform operations comprising:
extracting object/person re-identification (REID) embeddings from object/human crops by a teacher-student network;
using the REID embeddings as a supervision signal to train a REID branch for a neural network; and
processing the REID embeddings generated by the teacher-student network to track object/person identities across multiple images in a scene having object overlaps and occlusions.

### Claim 18 (depends on 17)

The system of claim 17, the operations comprising employing a memory-based online learning approach to update the REID embeddings as new video frames are processed.

### Claim 19 (depends on 17)

The system of claim 17, wherein the REID embeddings are used to generate a discriminative signature for each object/person, invariant to changes in camera position and lighting conditions.

### Claim 20

A computer readable medium that stores a set of instructions that is executable by at least one processor to cause the at least one processor to perform operations comprising:
extracting object/person re-identification (REID) embeddings from object/human crops by a teacher-student network;
using the REID embeddings as a supervision signal to train a REID branch for a neural network; and
processing the REID embeddings generated by the teacher-student network to track object/person identities across multiple images in a scene having object overlaps and occlusions.
