# US 2023/0065700 A1 · Data-driven extraction and composition of secondary dynamics in facial performance capture

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0065700 A1
- Document ID: US20230065700A1
- Status: Published application
- Kind: A1
- Application number: 18/045,782
- Filed: 2022-10-11
- Published: 2023-03-02
- Earliest claimed: 2019-12-03
- Assignees: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich)
- Applicants: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich)
- Inventors: Dominik Thabo BEELER; Derek Edward BRADLEY; Eftychios Dimitrios SIFAKIS; Gaspard ZOSS
- Web page: https://patentgrep.com/patent/US20230065700A1

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06T 13/80: Two-dimensional [2D] animation, e.g. using sprites
- G06V 40/176: {Dynamic expression}
- G06T 2215/06: Curved planar reformation of 3D line structures
- G06T 2215/16: Using real world measurements to influence rendering
- G06V 2201/033: of skeletal patterns
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A modeling engine generates a prediction model that quantifies and predicts secondary dynamics associated with the face of a performer enacting a performance. The modeling engine generates a set of geometric representations that represents the face of the performer enacting different facial expressions under a range of loading conditions. For a given facial expression and specific loading condition, the modeling engine trains a Machine Learning model to predict how soft tissue regions of the face of the performer change in response to external forces applied to the performer during the performance. The modeling engine combines different expression models associated with different facial expressions to generate a prediction model. The prediction model can be used to predict and remove secondary dynamics from a given geometric representation of a performance or to generate and add secondary dynamics to a given geometric representation of a performance.

## Claims

### Claim 1

A computer-implemented method for generating computer animations of facial expressions, the method comprising:
receiving training data associated with a performer enacting one or more facial expressions;
generating a first geometric model of a first facial expression included in the one or more facial expressions based on the training data;
generating a first set of values that correspond to one or more non-rigid portions of the face of the performer based on the first geometric model;
generating a second set of values that correspond to one or more rigid portions of the face of the performer based on the first geometric model; and
training a first expression model associated with the first facial expression based on the first set of values and the second set of values, wherein the first expression model is usable to predict an amount of secondary dynamics associated with the first facial expression.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein generating the first set of values comprises identifying a plurality of non-rigid points included in the first geometric model that correspond to the one or more non-rigid portions of the face of the performer.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein generating the first set of values further comprises comparing the plurality of non-rigid points with a plurality of points included in a default geometric model associated with the performer.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein generating the second set of values comprises identifying a plurality of rigid points included in the first geometric model that correspond to the one or more rigid portions of the face of the performer.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein generating the second set of values further comprises determining, for each rigid point included in the plurality of rigid points, one or more positions of the rigid point over an interval of time.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the first expression model is trained to receive a set of velocity values associated with one or more rigid portions of at least one face and generate a set of delta values associated with one or more non-rigid portions of the at least one face.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising generating a first prediction model based on the first expression model, wherein the first prediction model is configured to quantify different secondary dynamics for different facial expressions.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein generating the first prediction model comprises generating a set of blendshape weights based on the first prediction model.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating a second geometric model of a second facial expression included in the one or more facial expressions based on the training data;
generating a third set of values that correspond to the one or more non-rigid portions of the face of the performer based on the second geometric model;
generating a fourth set of values that correspond to the one or more rigid portions of the face of the performer based on the second geometric model; and
training a second expression model associated with the second facial expression based on the third set of values and the fourth set of values, wherein the second expression model is usable to predict an amount of secondary dynamics associated with the second facial expression.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, further comprising generating a first prediction model based on the first expression model and the second expression model, wherein the first prediction model is configured to quantify different secondary dynamics for different facial expressions.

### Claim 11

One or more non-transitory computer-readable medium including instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving training data associated with a performer enacting one or more facial expressions;
generating a first geometric model of a first facial expression included in the one or more facial expressions based on the training data;
generating a first set of values that correspond to one or more non-rigid portions of the face of the performer based on the first geometric model;
generating a second set of values that correspond to one or more rigid portions of the face of the performer based on the first geometric model; and
training a first expression model associated with the first facial expression based on the first set of values and the second set of values, wherein the first expression model is usable to predict an amount of secondary dynamics associated with the first facial expression.

### Claim 12 (depends on 11)

The one or more non-transitory computer-readable medium of claim 11, wherein generating the first set of values comprises identifying a plurality of non-rigid points included in the first geometric model that correspond to the one or more non-rigid portions of the face of the performer.

### Claim 13 (depends on 12)

The one or more non-transitory computer-readable medium of claim 12, wherein generating the first set of values further comprises comparing the plurality of non-rigid points with a plurality of points included in a default geometric model associated with the performer.

### Claim 14 (depends on 11)

The one or more non-transitory computer-readable medium of claim 11, wherein generating the second set of values comprises identifying a plurality of rigid points included in the first geometric model that correspond to the one or more rigid portions of the face of the performer.

### Claim 15 (depends on 14)

The one or more non-transitory computer-readable medium of claim 14, wherein generating the second set of values further comprises determining, for each rigid point included in the plurality of rigid points, one or more positions of the rigid point over an interval of time.

### Claim 16 (depends on 11)

The one or more non-transitory computer-readable medium of claim 11, wherein the first expression model predicts an amount of secondary dynamics associated with the first facial expression in a first dimension, further comprising training a second expression model based on the first set of values and the second set of values, wherein the second expression model is usable to predict an amount of secondary dynamics associated with the first facial expression in a second dimension.

### Claim 17 (depends on 11)

The one or more non-transitory computer-readable medium of claim 11, further comprising:
training a second expression model based on the training data, wherein the second expression model is usable to predict an amount of secondary dynamics associated with a second facial expression included in the one or more facial expressions; and
combining the first expression model and a second expression model to generate a first prediction model, wherein the first prediction model is configured to quantify different secondary dynamics for different facial expressions.

### Claim 18 (depends on 17)

The one or more non-transitory computer-readable medium of claim 17, wherein combining the first expression model and the second expression model comprises generating a first set of weights associated with the first expression model and a second set of weights associated with the second expression model.

### Claim 19 (depends on 17)

The one or more non-transitory computer-readable medium of claim 17, wherein combining the first expression model and the second expression model comprises generating one or more sets of weights based on the training data, wherein each set of weights is associated with a different facial region of the face of the performer.

### Claim 20

A system comprising:
one or more memories storing instructions; and
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
receive training data associated with a performer enacting one or more facial expressions;
generate a first geometric model of a first facial expression included in the one or more facial expressions based on the training data;
generate a first set of values that correspond to one or more non-rigid portions of the face of the performer based on the first geometric model;
generate a second set of values that correspond to one or more rigid portions of the face of the performer based on the first geometric model; and
train a first expression model associated with the first facial expression based on the first set of values and the second set of values, wherein the first expression model is usable to predict an amount of secondary dynamics associated with the first facial expression.
