# US 2022/0284679 A1 · Method and apparatus for constructing three-dimensional face mesh, device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284679 A1
- Document ID: US20220284679A1
- Status: Published application
- Kind: A1
- Application number: 17/751,496
- Filed: 2022-05-23
- Published: 2022-09-08
- Earliest claimed: 2020-07-16
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Xiangkai LIN; Sheng WANG
- Web page: https://patentgrep.com/patent/US20220284679A1

## CPC

- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 17/205: {Re-meshing}
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 7/64: of convexity or concavity
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 15/04: Texture mapping
- G06V 40/165: {using facial parts and geometric relationships}
- G06T 2200/04: involving 3D image data
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/20216: Image averaging
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30196: Human being; Person
- G06T 2207/30201: Face
- G06T 2210/56: Particle system, point based geometry or rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method, an apparatus and a storage medium for constructing a three-dimensional (3D) facial mesh using artificial intelligence is disclosed. The method includes: obtaining a facial point cloud of a target object; determining, through an expansion calculation, pixel coordinates on a facial texture image of the target object that correspond to 3D data points in the facial point cloud, as index information of the 3D data points; performing triangulation on pixels on the facial texture image to obtain triangulation information; constructing an initial 3D facial mesh according to the triangulation information, the index information, and the facial point cloud; determining a non-core region in the initial 3D facial mesh; smoothing the non-core region in the initial 3D facial mesh; and replacing the non-core region in the initial 3D facial mesh with the smoothed non-core region to obtain a 3D facial mesh of the target object.

## Claims

### Claim 1

A method for constructing a three-dimensional (3D) facial mesh, performed by a computing device, the method comprising:
obtaining a facial point cloud of a target object;
determining, through an expansion calculation, pixel coordinates on a facial texture image of the target object that correspond to 3D data points in the facial point cloud as index information of the 3D data points;
performing triangulation on pixels on the facial texture image to obtain triangulation information, the triangulation information being used for indicating triangular topologies corresponding to the pixels;
constructing an initial 3D facial mesh according to the triangulation information, the index information and the facial point cloud;
determining a non-core region in the initial 3D facial mesh, the non-core region being a non-facial region in the initial 3D facial mesh;
smoothing the non-core region in the initial 3D facial mesh to obtain a smoothed non-core region; and
replacing the non-core region in the initial 3D facial mesh with the smoothed non-core region to obtain a 3D facial mesh of the target object.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the determining the non-core region in the initial 3D facial mesh comprises:
determining non-facial key pixels in the facial texture image;
determining non-facial key 3D data points and triangular topologies corresponding to the non-facial key 3D data points based on the non-facial key pixels, the triangulation information and the index information; and
determining a region formed by the triangular topologies corresponding to the non-facial key 3D data points as the non-core region.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the determining the non-facial key pixels in the facial texture image comprises:
determining a convex hull corresponding to a core region in the facial texture image, the core region being a region covered by facial key pixels in the facial texture image;
enlarging the convex hull based on a dilation threshold to obtain a target region; and
determining pixels in a region other than the target region in the facial texture image as the non-facial key pixels.

### Claim 4 (depends on 1)

The method according to claim 1, before the determining, through the expansion calculation, the pixel coordinates on the facial texture image of the target object that correspond to the 3D data points in the facial point cloud as the index information of the 3D data points, the method further comprising:
performing grid downsampling on the facial point cloud to obtain a downsampled facial point cloud, wherein grid densities corresponding to a neck region and a non-neck region in the downsampled facial point cloud are different; and
the determining, through the expansion calculation, the pixel coordinates on the facial texture image of the target object that correspond to the 3D data points in the facial point cloud as the index information of the 3D data points comprises:
determining, through the expansion calculation, the pixel coordinates on the facial texture image of the target object that correspond to 3D data points in the downsampled facial point cloud as the index information of the 3D data points.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the performing grid downsampling on the facial point cloud to obtain the downsampled facial point cloud comprises:
defining a plurality of grids corresponding to the neck region in a 3D space according to a first grid density, and defining a plurality of grids corresponding to the non-neck region according to a second grid density;
fusing the facial point cloud into a plurality of grids in the 3D space; and
averaging, under a same coordinate dimension, 3D coordinates of 3D data points in the facial point cloud included in each grid to obtain the downsampled facial point cloud.

### Claim 6 (depends on 1)

The method according to claim 1, wherein the determining, through the expansion calculation, the pixel coordinates on the facial texture image of the target object that correspond to 3D data points in the facial point cloud as the index information of the 3D data points comprises:
obtaining 3D coordinates of the 3D data points in the facial point cloud in a 3D coordinate system; and
determining, according to the 3D coordinates of the 3D data points through a cylindrical expansion calculation, the pixel coordinates on the facial texture image that correspond to the 3D data points, as the index information.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising:
obtaining n face images corresponding to n head postures of the target object, n being a positive integer;
obtaining a mapping relationship between the 3D data points in the facial point cloud and pixels on each of the n face images;
respectively obtaining image regions corresponding to the head postures from the n face images to obtain n valid regions;
for an ith valid region in the n valid regions, obtaining target 3D data points corresponding to pixels in the ith valid region according to the mapping relationship, i being a positive integer less than or equal to n;
obtaining target pixels, which are on a regional texture image corresponding to the ith valid region, corresponding to the target 3D data points according to the index information;
rendering the target pixels to generate the regional texture image corresponding to the ith valid region, pixel values of the target pixels being determined according to pixel values of the pixels in the ith valid region; and
performing image fusion on the n regional texture images to generate the facial texture image of the target object.

### Claim 8 (depends on 7)

The method according to claim 7, wherein the respectively obtaining image regions corresponding to the head postures from the n face images to obtain the n valid regions comprises:
for the ith face image in the n face images, detecting facial key points in the ith face image; and
obtaining a valid region in the ith face image according to positions of the facial key points and a predefined rule.

### Claim 9 (depends on 7)

The method according to claim 7, after the performing image fusion on the n regional texture images to generate the facial texture image of the target object, the method further comprising:
obtaining according to the index information, pixel values of pixels, which correspond to the 3D data points, on the facial texture image; and
rendering the 3D facial mesh according to the pixel values of the 3D data points, to generate a textured 3D facial mesh.

### Claim 10

A computing device, comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement a method for constructing a three-dimensional (3D) facial mesh, the method comprising:
obtaining a facial point cloud of a target object;
determining, through an expansion calculation, pixel coordinates on a facial texture image of the target object that correspond to 3D data points in the facial point cloud as index information of the 3D data points;
performing triangulation on pixels on the facial texture image to obtain triangulation information, the triangulation information being used for indicating triangular topologies corresponding to the pixels;
constructing an initial 3D facial mesh according to the triangulation information, the index information and the facial point cloud;
determining a non-core region in the initial 3D facial mesh, the non-core region being a non-facial region in the initial 3D facial mesh;
smoothing the non-core region in the initial 3D facial mesh to obtain a smoothed non-core region; and
replacing the non-core region in the initial 3D facial mesh with the smoothed non-core region to obtain a 3D facial mesh of the target object.

### Claim 11 (depends on 10)

The computing device according to claim 10, wherein the determining the non-core region in the initial 3D facial mesh comprises:
determining non-facial key pixels in the facial texture image;
determining non-facial key 3D data points and triangular topologies corresponding to the non-facial key 3D data points based on the non-facial key pixels, the triangulation information and the index information; and
determining a region formed by the triangular topologies corresponding to the non-facial key 3D data points as the non-core region.

### Claim 12 (depends on 11)

The computing device according to claim 11, wherein the determining the non-facial key pixels in the facial texture image comprises:
determining a convex hull corresponding to a core region in the facial texture image, the core region being a region covered by facial key pixels in the facial texture image;
enlarging the convex hull based on a dilation threshold to obtain a target region; and
determining pixels in a region other than the target region in the facial texture image as the non-facial key pixels.

### Claim 13 (depends on 10)

The computing device according to claim 10, wherein before the determining, through the expansion calculation, the pixel coordinates on the facial texture image of the target object that correspond to the 3D data points in the facial point cloud as the index information of the 3D data points, the method further comprises:
performing grid downsampling on the facial point cloud to obtain a downsampled facial point cloud, wherein grid densities corresponding to a neck region and a non-neck region in the downsampled facial point cloud are different; and
the determining, through the expansion calculation, the pixel coordinates on the facial texture image of the target object that correspond to the 3D data points in the facial point cloud as the index information of the 3D data points comprises:
determining, through the expansion calculation, the pixel coordinates on the facial texture image of the target object that correspond to 3D data points in the downsampled facial point cloud as the index information of the 3D data points.

### Claim 14 (depends on 13)

The computing device according to claim 13, wherein the performing grid downsampling on the facial point cloud to obtain the downsampled facial point cloud comprises:
defining a plurality of grids corresponding to the neck region in a 3D space according to a first grid density, and defining a plurality of grids corresponding to the non-neck region according to a second grid density;
fusing the facial point cloud into a plurality of grids in the 3D space; and
averaging, under a same coordinate dimension, 3D coordinates of 3D data points in the facial point cloud included in each grid to obtain the downsampled facial point cloud.

### Claim 15 (depends on 10)

The computing device according to claim 10, wherein the determining, through the expansion calculation, the pixel coordinates on the facial texture image of the target object that correspond to 3D data points in the facial point cloud as the index information of the 3D data points comprises:
obtaining 3D coordinates of the 3D data points in the facial point cloud in a 3D coordinate system; and
determining, according to the 3D coordinates of the 3D data points through a cylindrical expansion calculation, the pixel coordinates on the facial texture image that correspond to the 3D data points, as the index information.

### Claim 16 (depends on 10)

The computing device according to claim 10, wherein the method further comprises:
obtaining n face images corresponding to n head postures of the target object, n being a positive integer;
obtaining a mapping relationship between the 3D data points in the facial point cloud and pixels on each of the n face images;
respectively obtaining image regions corresponding to the head postures from the n face images to obtain n valid regions;
for an ith valid region in the n valid regions, obtaining target 3D data points corresponding to pixels in the ith valid region according to the mapping relationship, i being a positive integer less than or equal to n;
obtaining target pixels, which are on a regional texture image corresponding to the ith valid region, corresponding to the target 3D data points according to the index information;
rendering the target pixels to generate the regional texture image corresponding to the ith valid region, pixel values of the target pixels being determined according to pixel values of the pixels in the ith valid region; and
performing image fusion on the n regional texture images to generate the facial texture image of the target object.

### Claim 17 (depends on 16)

The computing device according to claim 16, wherein the respectively obtaining image regions corresponding to the head postures from the n face images to obtain the n valid regions comprises:
for the ith face image in the n face images, detecting facial key points in the ith face image; and
obtaining a valid region in the ith face image according to positions of the facial key points and a predefined rule.

### Claim 18 (depends on 16)

The computing device according to claim 16, wherein after the performing image fusion on the n regional texture images to generate the facial texture image of the target object, the method further comprises:
obtaining according to the index information, pixel values of pixels, which correspond to the 3D data points, on the facial texture image; and
rendering the 3D facial mesh according to the pixel values of the 3D data points, to generate a textured 3D facial mesh.

### Claim 19

A non-transitory computer-readable storage medium, storing at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement a method for constructing a three-dimensional (3D) facial mesh, the method comprising:
obtaining a facial point cloud of a target object;
determining, through an expansion calculation, pixel coordinates on a facial texture image of the target object that correspond to 3D data points in the facial point cloud as index information of the 3D data points;
performing triangulation on pixels on the facial texture image to obtain triangulation information, the triangulation information being used for indicating triangular topologies corresponding to the pixels;
constructing an initial 3D facial mesh according to the triangulation information, the index information and the facial point cloud;
determining a non-core region in the initial 3D facial mesh, the non-core region being a non-facial region in the initial 3D facial mesh;
smoothing the non-core region in the initial 3D facial mesh to obtain a smoothed non-core region; and
replacing the non-core region in the initial 3D facial mesh with the smoothed non-core region to obtain a 3D facial mesh of the target object.

### Claim 20 (depends on 19)

The non-transitory computer-readable storage medium according to claim 19, wherein the determining the non-core region in the initial 3D facial mesh comprises:
determining non-facial key pixels in the facial texture image;
determining non-facial key 3D data points and triangular topologies corresponding to the non-facial key 3D data points based on the non-facial key pixels, the triangulation information and the index information; and
determining a region formed by the triangular topologies corresponding to the non-facial key 3D data points as the non-core region.
