# US 2023/0056800 A1 · Shape refinement of three-dimensional (3d) mesh reconstructed from images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0056800 A1
- Document ID: US20230056800A1
- Status: Published application
- Kind: A1
- Application number: 17/408,928
- Filed: 2021-08-23
- Published: 2023-02-23
- Earliest claimed: 2021-08-23
- Assignees: Sony Group Corporation
- Applicants: Sony Group Corporation
- Inventors: JIE HU; MOHAMMAD GHARAVI-ALKHANSARI
- Web page: https://patentgrep.com/patent/US20230056800A1

## CPC

- G06T 7/579: from motion
- G06T 17/205: {Re-meshing}
- G06T 5/30: Erosion or dilatation, e.g. thinning
- G06T 5/70: Denoising; Smoothing
- G06T 7/11: Region-based segmentation
- G06T 7/529: from texture
- G06T 15/503: {Blending, e.g. for anti-aliasing}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20192: Edge enhancement; Edge preservation
- G06T 2207/30196: Human being; Person
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An electronic device and method for shape refinement of a 3D mesh reconstructed from images is disclosed. A set of images of an object is acquired and used to estimate a first 3D mesh of a head portion of the object. A first set of operations is executed on the first 3D mesh to generate a second 3D mesh. The first set of operations includes a removal of one or more regions which are unneeded for head-shape estimation and/or a removal of one or more mesh artifacts associated with a 3D shape or a topology of the first 3D mesh. A 3D template mesh is processed to determine a set of filling patches which corresponds to a set of holes in the second 3D mesh. Based on the second 3D mesh and the set of filling patches, a hole filling operation is executed to generate a final 3D mesh.

## Claims

### Claim 1

An electronic device, comprising:
circuitry configured to:
acquire a set of images of an object;
estimate a first three-dimensional (3D) mesh of a head portion of the object, based on the acquired set of images;
execute a first set of operations on the estimated first 3D mesh to generate a second 3D mesh, wherein the executed first set of operations comprises one or more of:
a removal of one or more regions which are unneeded for head- shape estimation, and
a removal of one or more mesh artifacts associated with a 3D shape or a topology of the first 3D mesh;
process a 3D template mesh to determine a set of filling patches corresponding to a set of holes in the generated second 3D mesh; and
execute, based on the generated second 3D mesh and the determined set of filling patches, a hole filling operation to generate a final 3D mesh.

### Claim 2 (depends on 1)

The electronic device according to claim 1, wherein the one or more regions which are unneeded for the head-shape estimation include at least one of coded target meshes or hair meshes, and
wherein the one or more mesh artifacts associated with the 3D shape or the topology include a first mesh artifact in which one or more facets of the first 3D mesh include edges which are longer than a threshold.

### Claim 3 (depends on 1)

The electronic device according to claim 1, wherein the circuitry is further configured to:
determine a dihedral angle between a pair of adjoining edges of the first 3D mesh; and
determine the pair of adjoining edges as a second mesh artifact of the one or more mesh artifacts, based on whether the determined dihedral angle is above a threshold.

### Claim 4 (depends on 3)

The electronic device according to claim 3, wherein the removal of the second mesh artifact comprises at least one of a removal of the determined pair of edges or a relaxation of the determined pair of edges using a Laplacian smoothening operation.

### Claim 5 (depends on 1)

The electronic device according to claim 1, wherein the circuitry is further configured to refine a boundary associated with each of the set of holes in the generated second 3D mesh based on execution of an erosion operation over the boundary associated with each of the set of holes.

### Claim 6 (depends on 1)

The electronic device according to claim 1, wherein the circuitry is further configured to classify the set of holes in the generated second 3D mesh into one or more first holes and one or more second holes, and
wherein a number of vertices in a boundary of the one or more first holes is below a threshold and a number of vertices in a boundary of the one or more second holes is above a threshold.

### Claim 7 (depends on 6)

The electronic device according to claim 6, wherein the circuitry is further configured to fill the one or more first holes in the generated second 3D mesh based on a shape of a boundary associated with each of the one or more first holes.

### Claim 8 (depends on 6)

The electronic device according to claim 6, wherein the processing of the 3D template mesh comprises execution of:
a first operation to warp, for a non-hole region of the second 3D mesh, the 3D template mesh based on a shape of the second 3D mesh to obtain a warped template mesh, and
a second operation to update the warped template mesh with a first boundary corresponding to the warpage of the non-hole region.

### Claim 9 (depends on 8)

The electronic device according to claim 8, wherein the processing of the 3D template mesh further comprises execution of a mesh retopology operation to:
project the second 3D mesh to the updated template mesh to obtain a projected 3D mesh;
determine, from the updated template mesh, a set of triangles associated with a first set of vertices at the first boundary corresponding to the warpage of the non-hole region;
determine a set of candidate vertices associated with the determined set of triangles based on the second 3D mesh;
connect the determined set of candidate vertices to obtain a second boundary of a first filling patch of the set of filling patches; and
determine the first filling patch based on the obtained second boundary.

### Claim 10 (depends on 9)

The electronic device according to claim 9, wherein the execution of the hole filling operation comprises:
a combination of the set of filling patches with the projected 3D mesh to obtain a connected mesh; and
a filling-up of gaps between connecting parts of the connected mesh to obtain a re-topologized 3D mesh.

### Claim 11 (depends on 10)

The electronic device according to claim 10, wherein the circuitry is further configured to:
determine a set of adjacent boundary intervals based on the projected 3D mesh and the set of filling patches included in the connected mesh; and
connect the determined set of adjacent boundary intervals in a pairwise manner to fill-up the gaps with triangles.

### Claim 12 (depends on 10)

The electronic device according to claim 10, wherein the execution of the hole filling operation further comprises:
an alpha-blending operation on the re-topologized 3D mesh and the second 3D mesh to obtain an alpha-blended 3D mesh; and
a pyramid-blending operation on the alpha-blended 3D mesh and the 3D template mesh to obtain a pyramid-blended 3D mesh.

### Claim 13 (depends on 12)

The electronic device according to claim 12, wherein the circuitry is further configured to execute a remeshing operation on the pyramid-blended 3D mesh to generate the final 3D mesh.

### Claim 14

A method, comprising:
acquiring a set of images of an object;
estimating a first three-dimensional (3D) mesh of a head portion of the object, based on the acquired set of images;
executing a first set of operations on the estimated first 3D mesh to generate a second 3D mesh, wherein the executed first set of operations comprises one or more of:
a removal of one or more regions which are unneeded for head-shape estimation, and
a removal of one or more mesh artifacts associated with a 3D shape or a topology of the first 3D mesh;
processing a 3D template mesh to determine a set of filling patches corresponding to a set of holes in the generated second 3D mesh; and
executing, based on the generated second 3D mesh and the determined set of filling patches, a hole filling operation to generate a final 3D mesh.

### Claim 15 (depends on 14)

The method according to claim 14, further comprising classifying the set of holes in the generated second 3D mesh into one or more first holes and one or more second holes, and
wherein a number of vertices in a boundary of the one or more first holes is below a threshold and a number of vertices in a boundary of the one or more second holes is above a threshold.

### Claim 16 (depends on 15)

The method according to claim 15, further comprising filling the one or more first holes in the generated second 3D mesh based on a shape of a boundary associated with each of the one or more first holes.

### Claim 17 (depends on 15)

The method according to claim 15, wherein the processing of the 3D template mesh comprises execution of:
a first operation that comprises warping, for a non-hole region of the second 3D mesh, the 3D template mesh based on a shape of the second 3D mesh to obtain a warped template mesh, and
a second operation that comprises updating the warped template mesh with a boundary corresponding to the warpage of the non-hole region.

### Claim 18 (depends on 17)

The method according to claim 17, wherein the processing of the 3D template mesh further comprises execution of a mesh retopology operation that comprises:
projecting the second 3D mesh to the updated template mesh to obtain a projected 3D mesh;
determining, from the updated template mesh, a set of triangles associated with a first set of vertices at the boundary corresponding to the warpage of the non-hole region;
determining a set of candidate vertices associated with the determined set of triangles based on the second 3D mesh;
connecting the determined set of candidate vertices to obtain a boundary of a first filling patch of the set of filling patches; and
determining the first filling patch based on the obtained boundary.

### Claim 19 (depends on 18)

The method according to claim 18, wherein the execution of the hole filling operation comprises:
combining the set of filling patches with the projected 3D mesh to obtain a connected mesh; and
filling-up gaps between connecting parts of the connected mesh to obtain a re-topologized 3D mesh.

### Claim 20

A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by a processor in an electronic device, causes the electronic device to execute operations, the operations comprising:
acquiring a set of images of an object;
estimating a first three-dimensional (3D) mesh of a head portion of the object, based on the acquired set of images;
executing a first set of operations on the estimated first 3D mesh to generate a second 3D mesh, wherein the executed first set of operations comprises one or more of:
a removal of one or more regions which are unneeded for head-shape estimation, and
a removal of one or more mesh artifacts associated with a 3D shape or a topology of the first 3D mesh;
processing a 3D template mesh to determine a set of filling patches corresponding to a set of holes in the generated second 3D mesh; and
executing, based on the generated second 3D mesh and the determined set of filling patches, a hole filling operation to generate a final 3D mesh.
