# US 2023/0410447 A1 · View dependent three-dimensional morphable models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410447 A1
- Document ID: US20230410447A1
- Status: Published application
- Kind: A1
- Application number: 17/845,884
- Filed: 2022-06-21
- Published: 2023-12-21
- Earliest claimed: 2022-06-21
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Ke-Li CHENG; Anupama S; Kuang-Man HUANG; Chieh-Ming KUO; Avani RAO; Chiranjib CHOUDHURI; Michel Adib SARKIS; Ajit Deepak GUPTE; Ning BI
- Web page: https://patentgrep.com/patent/US20230410447A1

## CPC

- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 7/50: Depth or shape recovery
- G06T 7/75: {involving models}
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06T 2200/08: involving all processing steps from image acquisition to 3D model generation
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30201: Face
- G06T 2219/2021: Shape modification
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Systems and techniques are provided for generating a three-dimensional (3D) facial model. For example, a process can include obtaining at least one input image associated with a face. In some aspects, the process can include obtaining a pose for a 3D facial model associated with the face. In some examples, the process can include generating, by a machine learning model, the 3D facial model associated with the face. In some cases, one or more parameters associated with a shape component of the 3D facial model are conditioned on the pose. In some implementations, the 3D facial model is configured to vary in shape based on the pose for the 3D facial model associated with the face.

## Claims

### Claim 1

A method of generating a three-dimensional (3D) facial model, the method comprising:
obtaining at least one input image associated with a face;
obtaining a pose for a 3D facial model associated with the face; and
generating, by a machine learning model, the 3D facial model associated with the face, wherein one or more parameters associated with a shape component of the 3D facial model are conditioned on the pose.

### Claim 2 (depends on 1)

The method of claim 1, wherein the 3D facial model is configured to vary in shape based on the pose for the 3D facial model associated with the face.

### Claim 3 (depends on 1)

The method of claim 1, wherein the pose for the 3D facial model comprises a pitch, a yaw, and a roll.

### Claim 4 (depends on 1)

The method of claim 1, wherein the one or more parameters associated with the shape component of the 3D facial model comprise coefficients of the shape component of the 3D facial model.

### Claim 5 (depends on 4)

The method of claim 4, wherein the 3D facial model further comprises a mean face component and a facial expression component, and wherein the 3D facial model comprises a combination of the mean face component, the shape component, and the facial expression component.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
obtaining an additional pose for an additional 3D facial model associated with the face, wherein the additional pose for the additional 3D facial model associated with the face is different from the pose for the 3D facial model associated with the face; and
generating, by the machine learning model, the additional 3D facial model associated with the face, wherein at least one parameter of one or more additional parameters associated with a shape component of the additional 3D facial model is different from at least one parameter of the one or more parameters associated with the shape component of the 3D facial model.

### Claim 7 (depends on 6)

The method of claim 6, wherein the additional 3D facial model is generated based on the at least one input image associated with the face.

### Claim 8 (depends on 7)

The method of claim 7, wherein the 3D facial model associated with the face and the additional 3D facial model associated with the face comprise a common facial expression.

### Claim 9 (depends on 6)

The method of claim 6, further comprising obtaining at least one additional input image associated with the face, wherein the additional 3D facial model is generated based on the at least one additional input image associated with the face, wherein:
the at least one input image associated with the face is associated with a first facial expression;
the at least one additional input image associated with the face is associated with a second facial expression; and
the first facial expression and the second facial expression are different.

### Claim 10 (depends on 1)

The method of claim 1, wherein generating the 3D facial model comprises:
generating one or more facial image feature vectors based on the at least one input image associated with the face;
generating a pose feature vector based on the pose for the 3D facial model; and
generating, based on the one or more facial image feature vectors and the pose feature vector, a plurality of 3D facial model coefficients associated with the 3D facial model.

### Claim 11 (depends on 10)

The method of claim 10, wherein the at least one input image associated with the face comprises two or more partial images of the face.

### Claim 12 (depends on 11)

The method of claim 11, wherein the at least one input image associated with the face comprises an image of a left eye of the face, an image of a right eye of the face, and an image of a mouth of the face.

### Claim 13 (depends on 12)

The method of claim 12, wherein generating the one or more facial image feature vectors comprises:
generating a first facial image feature vector based on the image of the left eye of the face;
generating a second facial image feature vector based on the image of the right eye of the face; and
generating a third facial image feature vector based on the image of the mouth of the face.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
combining the first facial image feature vector with the pose feature vector;
combining the second facial image feature vector with the pose feature vector; and
combining the third facial image feature vector with the pose feature vector.

### Claim 15 (depends on 14)

The method of claim 14, wherein combining the first facial image feature vector, the second facial image feature vector, and the third facial image feature vector with the pose feature vector comprises:
performing an element-wise sum of the first facial image feature vector and the pose feature vector;
performing an element-wise sum of the second facial image feature vector and the pose feature vector; and
performing an element-wise sum of the third facial image feature vector and the pose feature vector.

### Claim 16

An apparatus for generating a 3D facial model, comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
obtain at least one input image associated with a face;
obtain a pose for a 3D facial model associated with the face; and
generate, by a machine learning model, the 3D facial model associated with the face, wherein one or more parameters associated with a shape component of the 3D facial model are conditioned on the pose.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the 3D facial model is configured to vary in shape based on the pose for the 3D facial model associated with the face.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein the pose for the 3D facial model comprises a pitch, a yaw, and a roll.

### Claim 19 (depends on 16)

The apparatus of claim 16, wherein the one or more parameters associated with the shape component of the 3D facial model comprise coefficients of the shape component of the 3D facial model.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the 3D facial model further comprises a mean face component and a facial expression component, and wherein the 3D facial model comprises a combination of the mean face component, the shape component, and the facial expression component.

### Claim 21 (depends on 16)

The apparatus of claim 16, wherein the at least one processor is configured to:
obtain an additional pose for an additional 3D facial model associated with the face, wherein the additional pose for the additional 3D facial model associated with the face is different from the pose for the 3D facial model associated with the face; and
generate, by the machine learning model, the additional 3D facial model associated with the face, wherein at least one parameter of one or more additional parameters associated with a shape component of the additional 3D facial model is different from at least one parameter of the one or more parameters associated with the shape component of the 3D facial model.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein the additional 3D facial model is generated based on the at least one input image associated with the face.

### Claim 23 (depends on 22)

The apparatus of claim 22, wherein the 3D facial model associated with the face and the additional 3D facial model associated with the face comprise a common facial expression.

### Claim 24 (depends on 21)

The apparatus of claim 21, wherein the at least one processor is configured to:
obtain at least one additional input image associated with the face, wherein the additional 3D facial model is generated based on the at least one additional input image associated with the face, wherein:
the at least one input image associated with the face is associated with a first facial expression;
the at least one additional input image associated with the face is associated with a second facial expression; and
the first facial expression and the second facial expression are different.

### Claim 25 (depends on 16)

The apparatus of claim 16, wherein the at least one processor is configured to:
generate one or more facial image feature vectors based on the at least one input image associated with the face;
generate a pose feature vector based on the pose for the 3D facial model; and
generate, based on the one or more facial image feature vectors and the pose feature vector, a plurality of 3D facial model coefficients associated with the 3D facial model.

### Claim 26 (depends on 25)

The apparatus of claim 25, wherein the at least one input image associated with the face comprises two or more partial images of the face.

### Claim 27 (depends on 26)

The apparatus of claim 26, wherein the at least one input image associated with the face comprises an image of a left eye of the face, an image of a right eye of the face, and an image of a mouth of the face.

### Claim 28 (depends on 27)

The apparatus of claim 27, wherein the at least one processor is configured to:
generate a first facial image feature vector based on the image of the left eye of the face;
generate a second facial image feature vector based on the image of the right eye of the face; and
generate a third facial image feature vector based on the image of the mouth of the face.

### Claim 29 (depends on 28)

The apparatus of claim 28, at least one processor is configured to:
combine the first facial image feature vector with the pose feature vector;
combine the second facial image feature vector with the pose feature vector; and
combine the third facial image feature vector with the pose feature vector.

### Claim 30 (depends on 29)

The apparatus of claim 29, wherein combining the first facial image feature vector, the second facial image feature vector, and the third facial image feature vector with the pose feature vector comprises:
performing an element-wise sum of the first facial image feature vector and the pose feature vector;
performing an element-wise sum of the second facial image feature vector and the pose feature vector; and
performing an element-wise sum of the third facial image feature vector and the pose feature vector.
