# US 2021/0233326 A1 · Methods and apparatus to generate photo-realistic three-dimensional models of a photographed environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233326 A1
- Document ID: US20210233326A1
- Status: Published application
- Kind: A1
- Application number: 15/734,486
- Filed: 2018-09-28
- Published: 2021-07-29
- Earliest claimed: 2018-09-28
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Wenlong Li; Yuri Shpalensky; Alex Gendelman; Maria Bortman; Asaf Shiloni
- Web page: https://patentgrep.com/patent/US20210233326A1

## CPC

- G06T 5/70: Denoising; Smoothing
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 7/593: from stereo images
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 7/74: {involving reference images or patches}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/10021: Stereoscopic video; Stereoscopic image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30196: Human being; Person
- G06T 2207/30221: Sports video; Sports image
- G06T 2207/30224: Ball; Puck
- G06T 2207/30228: Playing field
- G06T 2207/30232: Surveillance
- G06T 2219/2012: Colour editing, changing, or manipulating; Use of colour codes
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Methods and apparatus to generate photo-realistic three-dimensional models of a photographed environment are disclosed. An apparatus includes an object position calculator to determine a three-dimensional (3D) position of an object detected within a first image of an environment and within a second image of the environment. The apparatus further includes a 3D model generator to generate a 3D model of the environment based on the first image and the second image. The apparatus also includes a model integrity analyzer to detect a difference between the 3D position of the object and the 3D model. The 3D model generator automatically modifies the 3D model based on the difference in response to the difference satisfying a confidence threshold.

## Claims

### Claim 1

An apparatus comprising:
an object position calculator to determine a three-dimensional (3D) position of an object detected within a first image of an environment and within a second image of the environment, the first image captured by a first camera in a first location relative to the environment, the second image captured by a second camera in a second location relative to the environment different than the first location;
a 3D model generator to generate a 3D model of the environment based on the first image and the second image; and
a model integrity analyzer to detect a difference between the 3D position of the object and the 3D model, the 3D model generator to automatically modify the 3D model based on the difference.

### Claim 2 (depends on 1)

The apparatus as defined in claim 1, wherein the 3D position of the object defines at least one of a placement, an orientation, or a shape of the object within a real-world 3D space corresponding to the environment.

### Claim 3 (depends on 1)

The apparatus as defined in claim 1, wherein the 3D model generator is to generate the 3D model independently of the object position calculator determining the 3D position of the object.

### Claim 4 (depends on 1)

The apparatus as defined in claim 1, further including an object detection engine to detect the object in the first image of the environment using a deep learning object detection model.

### Claim 5 (depends on 1)

The apparatus as defined in claim 1, wherein, in response to the difference indicating a portion of the object is missing from the 3D model, the 3D model generator is to modify the 3D model by adding the missing portion.

### Claim 6 (depends on 5)

The apparatus as defined in claim 5, wherein the missing portion corresponds to an entirety of the object.

### Claim 7 (depends on 5)

The apparatus as defined in claim 5, wherein the 3D model generator is to add the missing portion by:
generating a point cloud representation of the missing portion based on a model of the object;
positioning the point cloud representation of the missing portion within the 3D model at a location corresponding to the 3D position of the object; and
coloring the point cloud representation of the missing portion based on pixel color information from at least one of the first image or the second image.

### Claim 8 (depends on 5)

The apparatus as defined in claim 5, wherein the object corresponds to a person, the object position calculator is to determine the 3D position of the object by determining positions of body parts of the person based on a deep learning skeleton detection model, and the missing portion of the object corresponds to a missing body part of the person.

### Claim 9 (depends on 1)

The apparatus as defined in claim 1, wherein, in response to the model integrity analyzer determining that a region of the 3D model corresponding to the position of the object includes a duplicate portion of the object, the 3D model generator is to modify the 3D model by removing the duplicate portion.

### Claim 10 (depends on 1)

The apparatus as defined in claim 1, wherein the first image is captured at a same time as the second image.

### Claim 11 (depends on 1)

The apparatus as defined in claim 1, wherein the 3D model generator is to automatically modify the 3D model in response to the difference satisfying a confidence threshold and to disregard the difference when the difference does not satisfy the confidence threshold.

### Claim 12 (depends on 1)

The apparatus as defined in claim 1, wherein the object position calculator is to determine the 3D position of the object based on a first calibration matrix for the first camera and a second calibration matrix for the second camera.

### Claim 13 (depends on 1)

The apparatus as defined in claim 1, wherein the first image is a first frame in a video stream captured by the first camera, the object position calculator to determine the 3D position of the object within the first image based on a previous image captured by the first camera before the first image, the previous image corresponding to a different frame in the video stream.

### Claim 14

A system comprising:
a first camera to capture a first image of an environment, the first camera in a first location relative to the environment;
a second camera to capture a second image of the environment, the second camera in a second location different than the first location;
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the at least one processor to:
determine a three-dimensional (3D) position of an object detected within the first image and within the second image;
generate a 3D model of the environment based on the first image and the second image;
detect a difference between the 3D position of the object and the 3D model; and
automatically modify the 3D model based on the difference.

### Claim 15 (depends on 14)

The system as defined in claim 14, wherein the instructions cause the at least one processor is to generate the 3D model independently of the at least one processor determining the 3D position of the object.

### Claim 16 (depends on 14)

The system as defined in claim 14, wherein the instructions further cause the at least one processor to, in response to the difference indicating a portion of the object is missing from the 3D model, modify the 3D model by adding the missing portion.

### Claim 17 (depends on 16)

The system as defined in claim 16, wherein the instructions cause the at least one processor to add the missing portion by:
generating a point cloud representation of the missing portion based on a model of the object;
positioning the point cloud representation of the missing portion within the 3D model at a location corresponding to the 3D position of the object; and
coloring the point cloud representation of the missing portion based on pixel color information from at least one of the first image or the second image.

### Claim 18 (depends on 16)

The system as defined in claim 16, wherein the instructions further cause the at least one processor to, in response to determining that a region of the 3D model corresponding to the position of the object includes a duplicate portion of the object, modify the 3D model by removing the duplicate portion.

### Claim 19

A non-transitory computer readable medium comprising instructions that, when executed, cause a machine to at least:
determine a three-dimensional (3D) position of an object detected within a first image of an environment and within a second image of the environment, the first image captured by a first camera in a first location relative to the environment, the second image captured by a second camera in a second location relative to the environment different than the first location;
generate a 3D model of the environment based on the first image and the second image;
detect a difference between the 3D position of the object and the 3D model; and
automatically modify the 3D model based on the difference.

### Claim 20 (depends on 19)

The non-transitory computer readable medium as defined in claim 19, wherein the instructions further cause the machine to generate the 3D model independently of the machine determining the 3D position of the object.

### Claim 21 (depends on 19)

The non-transitory computer readable medium as defined in claim 19, wherein the instructions further cause the machine to, in response to the difference indicating a portion of the object is missing from the 3D model, modify the 3D model by adding the missing portion.

### Claim 22 (depends on 19)

The non-transitory computer readable medium as defined in claim 19, wherein the instructions further cause the machine to, in response to determining that a region of the 3D model corresponding to the position of the object includes a duplicate portion of the object, modify the 3D model by removing the duplicate portion.

### Claim 23

A method comprising:
determining, by executing an instruction with at least one processor, a three-dimensional (3D) position of an object detected within a first image of an environment and within a second image of the environment, the first image captured by a first camera in a first location relative to the environment, the second image captured by a second camera in a second location relative to the environment different than the first location;
generating, by executing an instruction with the at least one processor, a 3D model of the environment based on the first image and the second image;
detecting, by executing an instruction with the at least one processor, a difference between the 3D position of the object and the 3D model; and
automatically modifying, by executing an instruction with the at least one processor, the 3D model based on the difference.

### Claim 24 (depends on 23)

The method as defined in claim 23, further including, in response to the difference indicating a portion of the object is missing from the 3D model, modifying the 3D model by adding the missing portion.

### Claim 25 (depends on 23)

The method as defined in claim 23, further including, in response to determining that a region of the 3D model corresponding to the position of the object includes a duplicate portion of the object, modifying the 3D model by removing the duplicate portion.
