# US 2020/0349763 A1 · Semantic Fusion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0349763 A1
- Document ID: US20200349763A1
- Status: Published application
- Kind: A1
- Application number: 16/403,421
- Filed: 2019-05-03
- Published: 2020-11-05
- Earliest claimed: 2019-05-03
- Assignees: Facebook Technologies, LLC
- Applicants: Facebook Technologies, LLC
- Inventors: Yu Fan Chen; Richard Andrew Newcombe; Lingni Ma
- Web page: https://patentgrep.com/patent/US20200349763A1

## CPC

- G06T 17/205: {Re-meshing}
- G06T 19/00: Manipulating three-dimensional [3D] models or images for computer graphics
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/174: involving the use of two or more images
- G06T 7/73: using feature-based methods
- G06T 2210/04: Architectural design, interior design
- G06T 2219/004: Annotating, labelling
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

In one embodiment, a computing system accesses a plurality of images captured by one or more cameras from a plurality of camera poses. The computing system generates, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images. The computing system accesses a three-dimensional (3D) model of the one or more objects. The computing system determines, using the plurality of camera poses, a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects. The computing system generates a semantic 3D model by projecting the semantic information of the plurality of semantic segmentations towards the 3D model using the plurality of virtual camera poses.

## Claims

### Claim 1

A method comprising, by a computing system:
accessing a plurality of images captured by one or more cameras from a plurality of camera poses;
generating, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images;
accessing a three-dimensional (3D) model of the one or more objects;
determining, using the plurality of camera poses, a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects;
generating a semantic 3D model by projecting the semantic information of the plurality of semantic segmentations towards the 3D model using the plurality of virtual camera poses.

### Claim 2 (depends on 1)

The method of claim 1, further comprising generating, using the plurality of images, a plurality of geometry-based segmentations comprising geometric information of the one or more objects captured in the plurality of images.

### Claim 3 (depends on 2)

The method of claim 2, wherein generating the semantic 3D model by projecting the semantic information of the plurality of semantic segmentations further comprises using the geometric information of the one or more objects to project the semantic information corresponding to the one or more objects captured in the plurality of images.

### Claim 4 (depends on 1)

The method of claim 1, further comprising generating, using the plurality of images, a plurality of instance segmentations comprising object identification of the one or more objects captured in the plurality of images.

### Claim 5 (depends on 4)

The method of claim 4, further comprising generating an instance 3D model by projecting the object identification of the plurality of instance segmentations towards the 3D model using the plurality of virtual camera poses.

### Claim 6 (depends on 5)

The method of claim 5, wherein generating the instance 3D model comprises combining object identification from each of the plurality of instance segmentations to apply to one of the one or more objects.

### Claim 7 (depends on 1)

The method of claim 1, further comprising generating a semantic tree based on the semantic information of the one or more objects captured in the plurality of images, wherein the semantic tree comprises a plurality of semantic levels, and wherein each semantic level corresponds to a threshold amount of semantic information.

### Claim 8 (depends on 7)

The method of claim 7, wherein the plurality of semantic levels comprises one or more of:
a first semantic level that comprises information corresponding to individual objects of the one or more objects,
a second semantic level that comprises information corresponding to object classes of the one or more objects, and
a third semantic level that comprises information corresponding to a semantic set of the one or more objects.

### Claim 9 (depends on 7)

The method of claim 7, wherein projecting the semantic information further comprises projecting the semantic information of a particular semantic level towards the 3D model.

### Claim 10 (depends on 1)

The method of claim 1, wherein the 3D model of the one or more objects is generated based on the plurality of images.

### Claim 11 (depends on 1)

The method of claim 1, further comprising calculating one or more confidence values that each semantic information of the one or more objects from the plurality of semantic segmentations corresponds to the respective object.

### Claim 12 (depends on 11)

The method of claim 11, wherein objects are associated with a plurality of environments, and wherein the one or more confidence values is based on whether the semantic information of the one or more objects corresponds to objects within a particular environment.

### Claim 13 (depends on 11)

The method of claim 11, wherein projecting the semantic information of the plurality of semantic segmentations towards the 3D model further comprises:
determining the confidence value of semantic information for one object exceeds a threshold confidence value; and
projecting the semantic information towards the 3D model of the respective object in response to the confidence value exceeding the threshold confidence value.

### Claim 14 (depends on 11)

The method of claim 11, wherein the semantic information associated with a confidence value less than a threshold confidence value is discarded.

### Claim 15 (depends on 1)

The method of claim 1, wherein generating the semantic 3D model comprises combining semantic information from each of the plurality of semantic segmentations to apply to one of the one or more objects.

### Claim 16 (depends on 1)

The method of claim 1, wherein projecting the semantic information towards the 3D model comprises adding a label that corresponds to the respective object in the 3D model.

### Claim 17

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access a plurality of images captured by one or more cameras from a plurality of camera poses;
generate, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images;
access a three-dimensional (3D) model of the one or more objects;
determine, using the plurality of camera poses, a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects;
generate a semantic 3D model by projecting the semantic information of the plurality of semantic segmentations towards the 3D model using the plurality of virtual camera poses.

### Claim 18

A system comprising: one or more processors; and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to:
access a plurality of images captured by one or more cameras from a plurality of camera poses;
generate, using the plurality of images, a plurality of semantic segmentations comprising semantic information of one or more objects captured in the plurality of images;
access a three-dimensional (3D) model of the one or more objects;
determine, using the plurality of camera poses, a corresponding plurality of virtual camera poses relative to the 3D model of the one or more objects;
generate a semantic 3D model by projecting the semantic information of the plurality of semantic segmentations towards the 3D model using the plurality of virtual camera poses.
