# US 2022/0012953 A1 · Method and system of rendering a 3d image for automated facial morphing with a learned generic head model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0012953 A1
- Document ID: US20220012953A1
- Status: Published application
- Kind: A1
- Application number: 17/338,509
- Filed: 2021-06-03
- Published: 2022-01-13
- Earliest claimed: 2018-04-30
- Inventors: Mathew Powers
- Web page: https://patentgrep.com/patent/US20220012953A1

## CPC

- G06T 7/344: {involving models}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06T 7/149: involving deformable models, e.g. active contour models
- G06T 15/04: Texture mapping
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06V 10/82: using neural networks
- G06V 40/162: {using pixel segmentation or colour matching}
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/166: {using acquisition arrangements}
- G06T 2207/10024: Color image
- G06T 2207/10028: Range image; Depth image; 3D point clouds
- G06T 2207/30201: Face
- G06T 2210/44: Morphing
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

In one aspect, a computerized method for rendering a three-dimensional (3D) digital image for automated facial morphing includes the step scanning of the user's face with a digital camera to obtain a set of digital images of the user's face. The method includes the step of determining that a user's face is in a compliant state. The method includes the step of implementing an analysis of the set of digital images and implementing a set of pre-rendering steps. Each digital image comprises a depth data, a red/green/blue (RGB) data, and a facemask data. The method then implements an iterative closest path (ICP) algorithm that correlates the set of digital images together by stitching together the cloud of points of the facemask data of each digital image and outputs a set of transformation matrices. The method includes the step of implementing a truncated signed distance function (TSDF) algorithm on the set of transformation matrices. The TSDF algorithm represents each point of the transformation matrices in a regularized voxel grid and outputs a set of voxel representations as a one-dimension (1-D) array of voxels. The method includes the step of implementing a marching cubes algorithm that obtains each voxel representation of the 1-D array of voxels and creates a three-dimensional (3D) mesh out of the per-voxel values provided by the TSDF and outputs a mesh representation. The mesh representation comprises a set of triangles and vertices. The method comprises the step of implementing a cleaning algorithm that obtains the mesh representation and cleans the floating vertices and triangles and outputs a mesh. The mesh comprises a set of scattered points with a normal per point. The method includes the step of implementing a Poisson algorithm on the mesh output and fills in any holes of the mesh. The Poisson algorithm outputs a reconstructed mesh. The method fits the reconstructed mesh on a trained three-dimensional (3D) face model and a specified machine learning algorithm is used to fit the trained 3D face model to the 3D landmarks in the reconstructed mesh.

## Claims

### Claim 1

A computerized method for rendering a three-dimensional (3D) digital image for automated facial morphing comprising:
scanning of a user's face with a digital camera to obtain a set of digital images of the user's face;
determining that the user's face is in a compliant state;
implementing an analysis of the set of digital images and implementing a set of pre-rendering steps by:
wherein each digital image comprises a depth data, a red/green/blue (RGB) data, and a facemask data,
implementing an iterative closest path (ICP) algorithm that correlates the set of digital images together by stitching together the cloud of points of the facemask data of each digital image and outputs a set of transformation matrices,
implementing a truncated signed distance function (TSDF) algorithm on the set of the transformation matrices, wherein the TSDF algorithm represents each point of the transformation matrices in a regularized voxel grid and outputs a set of voxel representations as a one-dimension (1-D) array of voxels,
implementing a marching cubes algorithm that obtains each voxel representation of the 1-D array of voxels and creates a three-dimensional (3D) mesh out of the per-voxel values provided by the TSDF and outputs a mesh representation, wherein the mesh representation comprises a set of triangles and vertices,
implementing a cleaning algorithm that obtains the mesh representations and cleans the floating vertices and triangles and outputs a mesh, wherein the mesh comprises a set of scattered points with a normal per point,
implementing a Poisson algorithm on the mesh output and fills in any holes of the mesh, wherein the Poisson algorithm outputs a reconstructed mesh,
fitting the reconstructed mesh on a trained three-dimensional (3D) face model, wherein a specified machine learning algorithm is used to fit the trained 3D face model to the 3D landmarks in the reconstructed mesh;
implementing a texture mapping algorithm that outputs a mask representation, wherein the mask representation illustrates each piece of the digital image with texture values and label values, and
implementing an image stitching algorithm that stitches the digital images with different lighting values to remove any seams; and outputs a single digital image with consistent lighting; and
rendering the 3D digital image of the user's face from the single digital image with consistent lighting.

### Claim 2 (depends on 1)

The computerized method of claim 1, wherein the trained three-dimensional (3D) face model comprises a FLAME (Faces Learned with an Articulated Model and Expressions) model.

### Claim 3 (depends on 2)

The computerized method of claim 2, wherein the machine learning comprises a Deep Neural Network algorithm.

### Claim 4 (depends on 3)

The computerized method of claim 3, wherein the machine learning comprises a TENSORFLOW algorithm.

### Claim 5 (depends on 4)

The computerized method of claim 4, wherein the TENSORFLOW algorithm is used to load and sample the FLAME model, and to fit the FLAME model to the 3D landmarks of the reconstructed mesh.

### Claim 6 (depends on 1)

The computerized method of claim 1, wherein the fitting of the reconstructed mesh to the FLAME model is performed on a cloud-computing based server.

### Claim 7 (depends on 1)

The computerized method of claim 1, wherein the fitting of the reconstructed mesh to the FLAME model is performed on a cloud-computing based server.

### Claim 8 (depends on 7)

The computerized method of claim 7, wherein the output of the cloud-computing based server is communicated to a mobile device for implementing the texture mapping algorithm.

### Claim 9 (depends on 1)

The computerized method of claim 1, wherein the fitting of the reconstructed mesh to the FLAME model is performed on the mobile device.

### Claim 10 (depends on 1)

The computerized method of claim 1, wherein the digital camera comprises a mobile-device digital camera array.

### Claim 11 (depends on 1)

The computerized method of claim 1, wherein the 3D digital image of the user's face is displayed on the mobile device.

### Claim 12 (depends on 5)

The computerized method of claim 5, wherein the fitting of the reconstructed mesh to the FLAME model is performed only fora face portion of the user's face.

### Claim 13 (depends on 12)

The computerized method of claim 12, wherein the FLAME model comprises a linear shape space trained from a specified number of scans of human heads.
