# US 10,769,848 B1 · 3D object reconstruction using photometric mesh representation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,848 B1
- Document ID: US10769848B1
- Status: Granted
- Kind: B1
- Application number: 16/421,729
- Filed: 2019-05-24
- Granted: 2020-09-08
- Earliest claimed: 2019-05-24
- Assignees: Adobe Inc.
- Applicants: Adobe, Inc.
- Inventors: Oliver Wang; Vladimir Kim; Matthew Fisher; Elya Shechtman; Chen-Hsuan Lin; Bryan Russell
- Web page: https://patentgrep.com/patent/US10769848B1

## CPC

- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- B29C 64/386: Data acquisition or data processing for additive manufacturing
- B33Y 50/00: Data acquisition or data processing for additive manufacturing
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06T 7/50: Depth or shape recovery
- G06T 7/55: from multiple images
- G06T 15/005: {General purpose rendering architectures}
- G06N 3/045: Combinations of networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 2200/04: involving 3D image data
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- B29C: SHAPING OR JOINING OF PLASTICS; SHAPING OF MATERIAL IN A PLASTIC STATE, NOT OTHERWISE PROVIDED FOR; AFTER-TREATMENT OF THE SHAPED PRODUCTS, e.g. REPAIRING (making preforms B29B11/00; making laminated products by combining previously unconnected layers which become one product whose layers will remain together B32B37/00 - B32B41/00)
- B33Y: ADDITIVE MANUFACTURING, i.e. MANUFACTURING OF THREE-DIMENSIONAL [3D] OBJECTS BY ADDITIVE DEPOSITION, ADDITIVE AGGLOMERATION OR ADDITIVE LAYERING, e.g. BY 3D PRINTING, STEREOLITHOGRAPHY OR SELECTIVE LASER SINTERING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Techniques are disclosed for 3D object reconstruction using photometric mesh representations. A decoder is pretrained to transform points sampled from 2D patches of representative objects into 3D polygonal meshes. An image frame of the object is fed into an encoder to get an initial latent code vector. For each frame and camera pair from the sequence, a polygonal mesh is rendered at the given viewpoints. The mesh is optimized by creating a virtual viewpoint, rasterized to obtain a depth map. The 3D mesh projections are aligned by projecting the coordinates corresponding to the polygonal face vertices of the rasterized mesh to both selected viewpoints. The photometric error is determined from RGB pixel intensities sampled from both frames. Gradients from the photometric error are backpropagated into the vertices of the assigned polygonal indices by relating the barycentric coordinates of each image to update the latent code vector.

## Claims

### Claim 1

In a digital medium environment for editing digital images, a computer-implemented method of three-dimensional object reconstruction, the method comprising:
generating, by at least one processor, a polygon mesh representing a shape of a first object in three dimensions using an object mesh generation neural network trained to transform a set of two-dimensional (2D) data points representing the first object into the polygon mesh, the set of 2D data points representing color pixels in at least two images of the first object, the at least two images having different camera poses;
optimizing by the at least one processor, the polygon mesh over a set of latent feature vectors using a photometric objective function to produce a reconstructed representation of the first object, the set of latent feature vectors representing a pre-defined shape of a second object; and
causing, by the at least one processor, the reconstructed representation of the first object to be output to an output device.

### Claim 2 (depends on 1)

The method of claim 1, further comprising training, by the at least one processor, the object mesh generation neural network to transform the set of 2D data points into the polygon mesh using 3D computer aided drawing (CAD) model renderings.

### Claim 3 (depends on 1)

The method of claim 1, wherein the second object is a model of the first object.

### Claim 4 (depends on 1)

The method of claim 1, wherein the photometric objective function represents, at least in part, a photometric loss contributed by pixels in the at least one face of the polygon mesh.

### Claim 5 (depends on 1)

The method of claim 1, wherein the photometric objective function includes applying a parameterized warp function to pixels in the at least two images of the first object.

### Claim 6 (depends on 1)

The method of claim 1, wherein the photometric objective function is:
ℒ
phot
(
j
)
⁡
(
V
j
)
=
∑
i
:
x
i
∈
𝒳
j
⁢

I
S
⁡
(
x
i
′
)
-
I
T
⁡
(
x
i
)

1
where x′i=(xi; Vj) is a warped pixel coordinate xi in a polygon mesh V of a first image IS of the first object, IT is a second image of the first object, V is the polygon mesh, and is a set of visible pixel coordinates within a projection range of the at least one face j of the polygon mesh.

### Claim 7 (depends on 1)

The method of claim 1, wherein the optimizing includes applying a scale penalty to the photometric objective function.

### Claim 8 (depends on 1)

The method of claim 1, wherein the output device includes at least one of a display device for displaying the reconstructed representation of the first object and a 3D printer device for 3D printing of the reconstructed representation of the first object.

### Claim 9 (depends on 1)

The method of claim 1, wherein the output device includes a memory device for storing the reconstructed representation of the first object.

### Claim 10

A computer program product including one or more non-transitory machine-readable mediums having instructions encoded thereon that when executed by at least one processor causes a process for 3D object reconstruction using photometric mesh representations to be carried out, the process comprising:
generating, by the at least one processor, a polygon mesh representing a shape of a first object in three dimensions using an object mesh generation neural network trained to transform a set of two-dimensional (2D) data points representing the first object into the polygon mesh, the set of 2D data points representing color pixels in at least two images of the first object, the at least two images having different camera poses;
optimizing, by the at least one processor, the polygon mesh over a set of latent feature vectors using a photometric objective function to produce a reconstructed representation of the first object, the set of latent feature vectors representing a pre-defined shape of a second object; and
causing by the at least one processor, the reconstructed representation of the first object to be at least one of displayed via a display device and printed via a 3D printing device.

### Claim 11 (depends on 10)

The computer program product of claim 10, wherein the process includes training, by the at least one processor, the object mesh generation neural network to transform the set of 2D data points into the polygon mesh using 3D computer aided drawing (CAD) model renderings.

### Claim 12 (depends on 10)

The computer program product of claim 10, wherein the photometric objective function represents, at least in part, a photometric loss contributed by pixels in the at least one face of the polygon mesh.

### Claim 13 (depends on 10)

The computer program product of claim 10, wherein the photometric objective function includes applying a parameterized warp function to pixels in the at least two images of the first object.

### Claim 14 (depends on 10)

The computer program product of claim 10, wherein the photometric objective function is:
ℒ
phot
(
j
)
⁡
(
V
j
)
=
∑
i
:
x
i
∈
𝒳
j
⁢

I
S
⁡
(
x
i
′
)
-
I
T
⁡
(
x
i
)

1
where x′i=(xi; Vj) is a warped pixel coordinate xi in a polygon mesh V of a first image IS of the first object, IT is a second image of the first object, V is the polygon mesh, and is a set of visible pixel coordinates within a projection range of the at least one face j of the polygon mesh.

### Claim 15

A system for 3D object reconstruction using photometric mesh representations, the system comprising:
at least one processor; and
a storage operatively coupled to the at least one processor and for storing instructions that when executed by the at least one processor cause the at least one processor to
generate a polygon mesh representing a shape of a first object in three dimensions using an object mesh generation neural network trained to transform a set of two-dimensional (2D) data points representing the first object into the polygon mesh, the set of 2D data points representing color pixels in at least two images of the first object, the at least two images having different camera poses;
optimize the polygon mesh over a set of latent feature vectors using a photometric objective function to produce a reconstructed representation of the first object, the set of latent feature vectors representing a pre-defined shape of a second object; and
cause the reconstructed representation of the first object to be at least one of displayed via a display device and printed via a 3D printing device.

### Claim 16 (depends on 15)

The system of claim 15, wherein the second object is a model of the first object.

### Claim 17 (depends on 15)

The system of claim 15, wherein the photometric objective function represents, at least in part, a photometric loss contributed by pixels in the at least one face of the polygon mesh.

### Claim 18 (depends on 15)

The system of claim 15, wherein the photometric objective function includes applying a parameterized warp function to pixels in the at least two images of the first object.

### Claim 19 (depends on 15)

The system of claim 15, wherein the photometric objective function is:
ℒ
phot
(
j
)
⁡
(
V
j
)
=
∑
i
:
x
i
∈
𝒳
j
⁢

I
S
⁡
(
x
i
′
)
-
I
T
⁡
(
x
i
)

1
where x′i=(xi; Vj) is a warped pixel coordinate xi in a polygon mesh V of a first image IS of the first object, IT is a second image of the first object, V is the polygon mesh, and is a set of visible pixel coordinates within a projection range of the at least one face j of the polygon mesh.

### Claim 20 (depends on 15)

The system of claim 15, wherein the optimizing includes applying a scale penalty to the photometric objective function.