# US 2023/0410404 A1 · Three dimensional object reconstruction for sensor simulation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410404 A1
- Document ID: US20230410404A1
- Status: Published application
- Kind: A1
- Application number: 18/209,609
- Filed: 2023-06-14
- Published: 2023-12-21
- Earliest claimed: 2022-06-15
- Assignees: WAABI Innovation Inc.
- Applicants: WAABI Innovation Inc.
- Inventors: Ioan Andrei Barsan; Yun Chen; Wei-Chiu Ma; Sivabalan Manivasagam; Raquel Urtasun; Jingkang Wang; Ze Yang
- Web page: https://patentgrep.com/patent/US20230410404A1

## CPC

- G06T 15/005: {General purpose rendering architectures}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 15/04: Texture mapping
- G06T 2219/2008: Assembling, disassembling
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Three dimensional object reconstruction for sensor simulation includes performing operations that include rendering, by a differential rendering engine, an object image from a target object model, and computing, by a loss function of the differential rendering engine, a loss based on a comparison of the object image with an actual image and a comparison of the target object model with a corresponding lidar point cloud. The operations further include updating the target object model by the differential rendering engine according to the loss, and rendering, after updating the target object model, a target object in a virtual world using the target object model.

## Claims

### Claim 1

A method comprising:
rendering, by a differential rendering engine, an object image from a target object model;
computing, by a loss function of the differential rendering engine, a loss based on a comparison of the object image with an actual image and a comparison of the target object model with a corresponding lidar point cloud;
updating the target object model by the differential rendering engine according to the loss; and
rendering, after updating the target object model, a target object in a virtual world using the target object model.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
obtaining an annotated CAD model;
obtaining a library CAD model for a target object;
deforming, by a CAD transformer engine, the annotated CAD model to match the library CAD model to generate a deformed annotated CAD model; and
annotating the library CAD model with an annotation from the deformed annotated CAD model to generate an annotated library model,
wherein the target object model is generated from the annotated library model.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
generating, with a parameterization engine, a decomposed object model from the annotated library model; and
storing the decomposed object model.

### Claim 4 (depends on 3)

The method of claim 3, wherein:
the decomposed object model comprises:
a first component model for a first component of the target object, and
a second component model for a second component of the target object,
the first component model and the second component model are individual and separate models, and
the first component model comprises a set of parameters detailing a connection between the first component model and the second component model.

### Claim 5 (depends on 3)

The method of claim 3, further comprising:
generating an object model from the decomposed object model, the decomposed object model comprising:
a first component model for a first component of the target object, and
a second component model for a second component of the target object,
wherein the first component model and the second component model are individual and separate models, and
wherein the first component model comprises a set of parameters detailing a connection between the first component model and the second component model.

### Claim 6 (depends on 5)

The method of claim 5, wherein the second component model is generic to a plurality of objects, and the set of object parameters comprises a location parameter detailing a placement of a second component in the first component and a scaling parameters detailing a scaling factor for the second component to fit the first component.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
calculating a loss as a combination of a data loss, a shape term, and an appearance term.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
calculating a color loss using the object image and the actual image;
calculating a LiDAR loss using a LiDAR point cloud and the target object model;
calculating a mask loss as a mask difference between an object mask of the object image and an actual mask of the actual image; and
calculating the data loss as a combination of the color loss, the LiDAR loss, and the mask loss.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
calculating the shape term using a normal consistency term and an edge length term.

### Claim 10 (depends on 7)

The method of claim 7, further comprising:
calculating the appearance term using the object image.

### Claim 11 (depends on 1)

The method of claim 1,
obtaining a first CAD model;
obtaining a library CAD model for a target object; and
deforming, by a CAD transformer engine, the first CAD model to match the library CAD model to generate a deformed model,
wherein the target object model is generated from the library CAD model, and
wherein rendering the target object in the virtual world comprises transferring a texture from the deformed model.

### Claim 12

A system comprising:
memory; and
at least one processor configured to execute instructions to perform operations comprising:
rendering, by a differential rendering engine, an object image from a target object model;
computing, by a loss function of the differential rendering engine, a loss based on a comparison of the object image with an actual image and a comparison of the target object model with a corresponding lidar point cloud;
updating the target object model by the differential rendering engine according to the loss; and
rendering, after updating the target object model, a target object in a virtual world using the target object model.

### Claim 13 (depends on 12)

The system of claim 12, wherein the operations further comprise:
obtaining an annotated CAD model;
obtaining a library CAD model for a target object;
deforming, by a CAD transformer engine, the annotated CAD model to match the library CAD model to generate a deformed annotated CAD model; and
annotating the library CAD model with an annotation from the deformed annotated CAD model to generate an annotated library model,
wherein the target object model is generated from the annotated library model.

### Claim 14 (depends on 13)

The system of claim 13, wherein the operations further comprise:
generating, with a parameterization engine, a decomposed object model from the annotated library model; and
store the decomposed object model.

### Claim 15 (depends on 14)

The system of claim 14, wherein:
the decomposed object model comprises:
a first component model for a first component of the target object, and
a second component model for a second component of the target object,
the first component model and the second component model are individual and separate models, and
the first component model comprises a set of parameters detailing a connection between the first component model and the second component model.

### Claim 16 (depends on 14)

The system of claim 14, wherein the operations further comprise:
generating an object model from the decomposed object model, the decomposed object model comprising:
a first component model for a first component of the target object, and
a second component model for a second component of the target object,
wherein the first component model and the second component model are individual and separate models, and
wherein the first component model comprises a set of parameters detailing a connection between the first component model and the second component model.

### Claim 17 (depends on 12)

The system of claim 12, wherein the operations further comprise:
calculating a loss as a combination of a data loss, a shape term, and an appearance term.

### Claim 18 (depends on 17)

The system of claim 17, wherein the operations further comprise:
calculating a color loss using the object image and the actual image;
calculating a LiDAR loss using a LiDAR point cloud and the target object model;
calculating a mask loss as a mask difference between an object mask of the object image and an actual mask of the actual image; and
calculating the data loss as a combination of the color loss, the LiDAR loss, and the mask loss.

### Claim 19

A non-transitory computer readable medium comprising computer readable program code for causing a computing system to perform operations comprising:
rendering, by a differential rendering engine, an object image from a target object model;
computing, by a loss function of the differential rendering engine, a loss based on a comparison of the object image with an actual image and a comparison of the target object model with a corresponding lidar point cloud;
updating the target object model by the differential rendering engine according to the loss; and
rendering, after updating the target object model, a target object in a virtual world using the target object model.

### Claim 20 (depends on 19)

The non-transitory computer readable medium of claim 19, further comprising:
obtaining an annotated CAD model;
obtaining a library CAD model for a target object;
deforming, by a CAD transformer engine, the annotated CAD model to match the library CAD model to generate a deformed annotated CAD model; and
annotating the library CAD model with an annotation from the deformed annotated CAD model to generate an annotated library model,
wherein the target object model is generated from the annotated library model.