# US 11,908,095 B2 · 2-D image reconstruction in a 3-D simulation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,095 B2
- Document ID: US11908095B2
- Status: Granted
- Kind: B2
- Application number: 17/527,867
- Filed: 2021-11-16
- Published: 2023-05-18
- Granted: 2024-02-20
- Earliest claimed: 2021-11-16
- Assignees: GM Cruise Holdings LLC
- Applicants: GM Cruise Holdings LLC
- Inventors: Ignacio Martin-Bragado
- Web page: https://patentgrep.com/patent/US11908095B2

## CPC

- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06F 16/587: using geographical or spatial information, e.g. location
- G06T 7/50: Depth or shape recovery
- G06T 7/66: of image moments or centre of gravity
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 15/20: Perspective computation
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 19/003: {Navigation within 3D models or images}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30252: Vehicle exterior; Vicinity of vehicle
- G06T 2219/2004: Aligning objects, relative positioning of parts
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present technology is directed to presenting a 3-D representation of an object, which is captured in a 2-D image of the object in an environment in a 3-D simulation of the environment. The present technology can receive 3-D data representing the environment including image data of the object in the environment, and a label identifying the object. The present technology can further locate a position of the object in the 3-D simulation of the environment based on determining a depth and an orientation of the object in the environment based on a semantic database of attributes associated with the object, obtain a 3-D representation of the object from the semantic database of attributes associated with the object, and project the 3-D representation of the object into the 3-D simulation of the environment at the determined position for the object.

## Claims

### Claim 1

A computer-implemented method for presenting a 3-D representation of an object, which was captured in a 2-D image of the object in an environment in a 3-D simulation of the environment, the method comprising:
receiving 3-D data representing the environment, the 3-D data representing the environment including a bounding box showing an approximate location of the object in the environment, image data of a visual representation of the object in the environment, and a label identifying the object;
locating a position of the object in the 3-D simulation of the environment created from the received 3-D data representing the environment, the locating the position of the object in the 3-D simulation of the environment including:
determining an average measurement of the object, a depth of the object in the environment, and an orientation of the object in the environment based on a semantic database of attributes associated with the object, wherein the determining the average measurement of the object, the depth of the object in the environment, and the orientation of the object in the environment based on the semantic database of attributes associated with the object includes determining the average measurement of the object based on a value stored in the semantic database of attributes associated with the object, determining the depth of the object in the environment based on a comparison of the average measurement of the object and an apparent measurement of the object in the 2-D image, and determining the orientation of the object in the environment based on a type of the object and the approximate location of the object in the environment;
obtaining a 3-D representation of the object from the semantic database of attributes associated with the object; and
projecting the 3-D representation of the object into the 3-D simulation of the environment at the determined position for the object.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
training an object identification model based on the 3-D representation of the object.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the determining the orientation of the object in the environment comprises:
determining that the object is oriented in a direction in-line with traffic when the object occurs within a road boundary in the environment.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the determining the orientation of the object in the environment comprises:
determining that the object is located outside of a road boundary in the environment; and
determining an orientation heuristic associated with the object in the semantic database of attributes associated with the object, wherein the orientation of the object is determined by the orientation heuristic.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the determining the orientation of the object in the environment comprises:
determining that the object is located outside of a road boundary in the environment; and
determining an orientation attribute associated with the object in the semantic database of attributes associated with the object,
wherein the orientation attribute defines an approximate orientation of the object with respect to the road boundary.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the semantic database of attributes associated with the object includes a set of assumptions and contextual information that provide an approximate value for the depth and the orientation for a type of the object.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining if the 3-D representation of the object projected into the 3-D simulation of the environment overlaps with another object located in the 3-D simulation of the environment; and
determining if a centroid of the 3-D representation of the object projected into the 3-D simulation of the environment is within a region, wherein the region is defined based on a set of assumptions and contextual information for a type of the object stored in the semantic database of attributes associated with the object.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, further comprising:
in response to a determination that the 3-D representation of the object overlaps with the another object or a determination that the centroid of the 3-D representation of the object is not within the region, adjusting the depth of the object by a predetermined increment or decrement.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the locating the position of the object in the 3-D simulation of the environment is based on 3-D image data of a 3-D object included in the 3-D data representing the environment.

### Claim 10

A system for presenting a 3-D representation of an object captured in a 2-D image of the object in an environment in a 3-D simulation of the environment, the system comprising:
a storage configured to store instructions; and
a processor configured to execute the instructions and cause the processor to:
receive 3-D data representing the environment, the 3-D data representing the environment including a bounding box showing an approximate location of the object in the environment, image data of a visual representation of the object in the environment, and a label identifying the object,
locate a position of the object in the 3-D simulation of the environment created from the received 3-D data representing the environment, the locate the position of the object in the 3-D simulation of the environment including,
determine an average measurement of the object, a depth of the object in the environment, and an orientation of the object in the environment based on a semantic database of attributes associated with the object,
determine the average measurement of the object based on a value stored in the semantic database of attributes associated with the object,
determine the depth of the object in the environment based on a comparison of the average measurement of the object and an apparent measurement of the object in the 2-D image,
determine the orientation of the object in the environment based on a type of the object and the approximate location of the object in the environment,
obtain a 3-D representation of the object from the semantic database of attributes associated with the object, and
project the 3-D representation of the object into the 3-D simulation of the environment at the determined position for the object.

### Claim 11 (depends on 10)

The system of claim 10, wherein the processor is configured to execute the instructions and cause the processor to:
train an object identification model based on the 3-D representation of the object.

### Claim 12 (depends on 10)

The system of claim 10, wherein the processor is configured to execute the instructions and cause the processor to:
determine if the 3-D representation of the object projected into the 3-D simulation of the environment overlaps with another object located in the 3-D simulation of the environment; and
determine if a centroid of the 3-D representation of the object projected into the 3-D simulation of the environment is within a region, wherein the region is defined based on a set of assumptions and contextual information for a type of the object stored in the semantic database of attributes associated with the object.

### Claim 13 (depends on 12)

The system of claim 12, wherein the processor is configured to execute the instructions and cause the processor to:
in response to a determination that the 3-D representation of the object overlaps with the another object or a determination that the centroid of the 3-D representation of the object is not within the region, adjust the depth of the object by a predetermined increment or decrement.

### Claim 14

A non-transitory computer readable medium for presenting a 3-D representation of an object captured in a 2-D image of the object in an environment in a 3-D simulation of the environment, the non-transitory computer readable medium comprising instructions, the instructions, when executed by a computing system, cause the computing system to:
receive 3-D data representing the environment, the 3-D data representing the environment including a bounding box showing an approximate location of the object in the environment, image data of a visual representation of the object in the environment, and a label identifying the object;
locate a position of the object in the 3-D simulation of the environment created from the received 3-D data representing the environment, the locate the position of the object in the 3-D simulation of the environment including;
determine an average measurement of the object, a depth of the object in the environment, and an orientation of the object in the environment based on a semantic database of attributes associated with the object;
determine the average measurement of the object based on a value stored in the semantic database of attributes associated with the object;
determine the depth of the object in the environment based on a comparison of the average measurement of the object and an apparent measurement of the object in the 2-D image;
determine the orientation of the object in the environment based on a type of the object and the approximate location of the object in the environment;
obtain a 3-D representation of the object from the semantic database of attributes associated with the object; and
project the 3-D representation of the object into the 3-D simulation of the environment at the determined position for the object.

### Claim 15 (depends on 14)

The computer readable medium of claim 14, wherein the computer readable medium further comprises instructions that, when executed by the computing system, cause the computing system to:
train an object identification model based on the 3-D representation of the object.

### Claim 16 (depends on 14)

The computer readable medium of claim 14, wherein the computer readable medium further comprises instructions that, when executed by the computing system, cause the computing system to:
determine if the 3-D representation of the object projected into the 3-D simulation of the environment overlaps with another object located in the 3-D simulation of the environment; and
determine if a centroid of the 3-D representation of the object projected into the 3-D simulation of the environment is within a region, wherein the region is defined based on a set of assumptions and contextual information for a type of the object stored in the semantic database of attributes associated with the object.

### Claim 17 (depends on 16)

The computer readable medium of claim 16, wherein the computer readable medium further comprises instructions that, when executed by the computing system, cause the computing system to:
in response to a determination that the 3-D representation of the object overlaps with the another object or a determination that the centroid of the 3-D representation of the object is not within the region, adjust the depth of the object by a predetermined increment or decrement.