# US 2025/0095259 A1 · Avatar animation with general pretrained facial movement encoding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095259 A1
- Document ID: US20250095259A1
- Status: Published application
- Kind: A1
- Application number: 18/470,032
- Filed: 2023-09-19
- Published: 2025-03-20
- Earliest claimed: 2023-09-19
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Ke-Li CHENG; Wei-Lun CHANG; Michel Adib SARKIS
- Web page: https://patentgrep.com/patent/US20250095259A1

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/778: Active pattern-learning, e.g. online learning of image or video features
- G06V 10/82: using neural networks
- G06V 40/174: {Facial expression recognition}
- G06V 40/176: {Dynamic expression}
- G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Techniques and systems are provided for generating a representation of a face. For instance, a process can include obtaining one or more images of a face. The process can further include generating an encoded expression representing an expression of the face, wherein predetermined characteristics of the face remain constant relative to the encoded expression. The process can further include mapping the encoded expression to a corresponding expression of a facial model. The process can further include generating the representation of the facial model based on the encoded expression.

## Claims

### Claim 1

A method for generating a representation of a face, the method comprising:
obtaining one or more images of a face;
generating an encoded expression representing an expression of the face, wherein predetermined characteristics of the face remain constant relative to the encoded expression;
mapping the encoded expression to a corresponding expression of a facial model; and
generating the representation of the facial model based on the encoded expression.

### Claim 2 (depends on 1)

The method of claim 1, wherein the encoded expression is based on motion features determined based on images of the face.

### Claim 3 (depends on 1)

The method of claim 1, wherein the predetermined characteristics of the face include at least one of a view angle, color style, or identity of the face.

### Claim 4 (depends on 1)

The method of claim 1, wherein the generating of the representation of the facial model is enhanced based on an audio signal obtained concurrently with the one or more images of the face.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
receiving a frame, the frame including at least a portion of a face;
encoding motion features of the frame into the encoded expression; and
outputting the encoded expression for transmission.

### Claim 6

A method for training an expression encoder, comprising:
obtaining a first frame and a second frame, the first frame and second frame including at least a portion of a face;
generating a first expression feature for the first frame, the first expression feature representing a first expression of the face;
generating a second expression feature for the second frame, the second expression feature representing a second expression of the face;
generating a first view angle feature for the first frame, the first view angle feature representing a first angle from which the face is viewed from;
generating a second view angle feature for the second frame, the second view angle feature representing a second angle from which the face is viewed from;
crossing at least one of one of the first expression feature and second expression feature or the first view angle feature and the second view angle feature;
determining a first loss value based on the crossing; and
adjusting a feature encoder based on the determined first loss value.

### Claim 7 (depends on 6)

The method of claim 6, wherein the first expression matches the second expression, and wherein the first view angle feature is crossed with the second view angle feature, and further comprising:
generating a first reconstructed image based on the crossed first view angle feature;
generating a second reconstructed image based on the crossed second view angle feature;
determining the first loss value based on a comparison between the first reconstructed image and the first frame; and
determining a second loss value based on a comparison between the second reconstructed image and the second frame.

### Claim 8 (depends on 7)

The method of claim 7, further comprising determining a third loss value based on the first expression feature and the second expression feature.

### Claim 9 (depends on 6)

The method of claim 6, wherein the first angle matches the second angle, and wherein the first expression feature is crossed with the second expression feature, and further comprising:
generating a first reconstructed image based on the crossed first expression feature;
generating a second reconstructed image based on the crossed second expression feature;
determining the first loss value based on a comparison between the first reconstructed image and the first frame; and
determining a second loss value based on a comparison between the second reconstructed image and the second frame.

### Claim 10 (depends on 9)

The method of claim 9, further comprising determining a third loss value based on the first view angle feature and the second view angle feature.

### Claim 11 (depends on 6)

The method of claim 6, further comprising:
augmenting the first frame to generate an augmented frame;
generating an augmented expression feature based on the augmented frame;
generating an augmented view angle feature based on the augmented frame;
obtaining a semantic labelled version of the first frame;
generating a surrogate expression feature based on the semantically labelled version of the first frame;
generating a surrogate view angle feature based on the semantically labelled version of the first frame; and
generating a third loss value based on a comparison between the augmented expression feature and the surrogate expression feature and a comparison between augmented view angle feature and the surrogate view angle feature.

### Claim 12 (depends on 11)

The method of claim 11, wherein augmenting the first frame comprises adjusting color channels of the first frame.

### Claim 13

An apparatus for generating a representation of a face, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor being configured to:
obtain one or more images of a face;
generate an encoded expression representing an expression of the face, wherein predetermined characteristics of the face remain constant relative to the encoded expression;
map the encoded expression to a corresponding expression of a facial model; and
generate the representation of the facial model based on the encoded expression.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the encoded expression is based on motion features determined based on images of the face.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein the predetermined characteristics of the face include at least one of a view angle, color style, or identity of the face.

### Claim 16 (depends on 13)

The apparatus of claim 13, wherein the generating of the representation of the facial model is enhanced based on an audio signal obtained concurrently with the one or more images of the face.

### Claim 17 (depends on 13)

The apparatus of claim 13, wherein the processor is further configured to:
receive a frame, the frame including at least a portion of a face;
encode motion features of the frame into the encoded expression; and
output the encoded expression for transmission.

### Claim 18

An apparatus for training an expression encoder, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor being configured to:
obtain a first frame and a second frame, the first frame and second frame including at least a portion of a face;
generate a first expression feature for the first frame, the first expression feature representing a first expression of the face;
generate a second expression feature for the second frame, the second expression feature representing a second expression of the face;
generate a first view angle feature for the first frame, the first view angle feature representing a first angle from which the face is viewed from;
generate a second view angle feature for the second frame, the second view angle feature representing a second angle from which the face is viewed from;
cross at least one of one of the first expression feature and second expression feature or the first view angle feature and the second view angle feature;
determine a first loss value based on the crossing; and
adjust a feature encoder based on the determined first loss value.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the first expression matches the second expression, and wherein the first view angle feature is crossed with the second view angle feature, and further comprising:
generate a first reconstructed image based on the crossed first view angle feature;
generate a second reconstructed image based on the crossed second view angle feature;
determine the first loss value based on a comparison between the first reconstructed image and the first frame; and
determine a second loss value based on a comparison between the second reconstructed image and the second frame.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the at least one processor is further configured to determine a third loss value based on the first expression feature and the second expression feature.

### Claim 21 (depends on 18)

The apparatus of claim 18, wherein the first angle matches the second angle, and wherein the first expression feature is crossed with the second expression feature, and wherein the at least one processor is further configured to:
generate a first reconstructed image based on the crossed first expression feature;
generate a second reconstructed image based on the crossed second expression feature;
determine the first loss value based on a comparison between the first reconstructed image and the first frame; and
determine a second loss value based on a comparison between the second reconstructed image and the second frame.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein the at least one processor is further configured to determine a third loss value based on the first view angle feature and the second view angle feature.

### Claim 23 (depends on 18)

The apparatus of claim 18, wherein the at least one processor is further configured to:
augment the first frame to generate an augmented frame;
generate an augmented expression feature based on the augmented frame;
generate an augmented view angle feature based on the augmented frame;
obtain a semantic labelled version of the first frame;
generate a surrogate expression feature based on the semantically labelled version of the first frame;
generate a surrogate view angle feature based on the semantically labelled version of the first frame; and
generate a third loss value based on a comparison between the augmented expression feature and the surrogate expression feature and a comparison between augmented view angle feature and the surrogate view angle feature.

### Claim 24 (depends on 23)

The apparatus of claim 23, wherein, to augment the first frame, the at least one processor is configured to adjust color channels of the first frame.

### Claim 25

A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to:
obtain one or more images of a face;
generate an encoded expression representing an expression of the face, wherein predetermined characteristics of the face remain constant relative to the encoded expression;
map the encoded expression to a corresponding expression of a facial model; and
generate a representation of the facial model based on the encoded expression.

### Claim 26 (depends on 25)

The non-transitory computer-readable medium of claim 25, wherein the encoded expression is based on motion features determined based on images of the face.

### Claim 27 (depends on 25)

The non-transitory computer-readable medium of claim 25, wherein the predetermined characteristics of the face include at least one of a view angle, color style, or identity of the face.

### Claim 28 (depends on 25)

The non-transitory computer-readable medium of claim 25, wherein the generating of the representation of the face from the one or more images is enhanced based on an audio signal obtained concurrently with the one or more images of the face.

### Claim 29 (depends on 25)

The non-transitory computer-readable medium of claim 25, wherein the instructions further cause the at least one processor to:
receive a frame, the frame including at least a portion of a face;
encode motion features of the frame into the encoded expression, wherein the encoded expression is face and view angle invariant; and
output the encoded expression for transmission.

### Claim 30

A non-transitory computer-readable medium having stored thereon instructions that, when executed by at least one processor, cause the at least one processor to:
obtain a first frame and a second frame, the first frame and second frame including at least a portion of a face;
generate a first expression feature for the first frame, the first expression feature representing a first expression of the face;
generate a second expression feature for the second frame, the second expression feature representing a second expression of the face;
generate a first view angle feature for the first frame, the first view angle feature representing a first angle from which the face is viewed from;
generate a second view angle feature for the second frame, the second view angle feature representing a second angle from which the face is viewed from;
cross at least one of one of the first expression feature and second expression feature or the first view angle feature and the second view angle feature;
determine a first loss value based on the crossing; and
adjust a feature encoder based on the determined first loss value.
