# US 11,443,473 B2 · Systems and methods for generating a skull surface for computer animation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,443,473 B2
- Document ID: US11443473B2
- Status: Granted
- Kind: B2
- Application number: 17/347,437
- Filed: 2021-06-14
- Published: 2022-03-24
- Granted: 2022-09-13
- Earliest claimed: 2020-09-18
- Assignees: Unity Technologies SF
- Applicants: Weta Digital Limited
- Inventors: Byung Kuk Choi
- Web page: https://patentgrep.com/patent/US11443473B2

## CPC

- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06F 16/53: Querying
- G06F 18/2413: based on distances to training or reference patterns
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 7/50: Depth or shape recovery
- G06T 15/205: {Image-based rendering}
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 40/165: {using facial parts and geometric relationships}
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G06V 40/174: {Facial expression recognition}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30201: Face
- G06T 2215/16: Using real world measurements to influence rendering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An animation system wherein a machine learning model is adopted to generate animated facial actions based on parameters obtained from a live actor. Specifically, the anatomical structure such as a facial muscle topology and a skull surface that are specific to the live actor may be used. A skull surface that is specific to a live actor based on facial scans of the live actor and generic tissue depth data. For example, the facial scans of the live actor may provide a skin surface topology of the live actor, based on which the skull surface underneath the skin surface can be derived by “offsetting” the skin surface with corresponding soft tissue depth at different sampled points on the skin surface.

## Claims

### Claim 1

A computer-implemented method for generating a bone surface depicting at least a portion of a skull of a live actor, the method comprising:
obtaining a plurality of facial scans of the live actor, each facial scan including a respective skin surface and respective sensing data indicative of facial muscle strains corresponding to a set of facial muscles;
obtaining, from a database, a tissue depth dataset including a tissue depth indicating a distance from a corresponding surface point on the human face to a skull surface underneath the human face;
deriving a bone surface based on a skin surface generated from the plurality of facial scans and the tissue depth dataset;
refining the bone surface based on a user input via a user interface; and
refining the bone surface via a neural network by inputting to the neural network the plurality of facial scans and a tissue depth array from the tissue depth dataset, and using muscle or skin parameters derived from the plurality of facial scans as ground truth labels.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
determining, from the plurality of facial scans, the three-dimensional facial skin surface structure of the live actor by:
determining, for a sampled point on the three-dimensional facial skin surface structure, a corresponding point on the bone surface based on a respective tissue depth corresponding to the sampled point; and
determining the bone surface by interpolating a plurality of determined corresponding points on the bone surface.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
generating a muscle model from the plurality of the facial scans of the live actor, wherein the muscle model comprises a plurality of facial muscles and each facial muscle includes an attachment point to the skull; and
refining the bone surface according to a set of attachment points indicated by the generated muscle model.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
receiving, via the user interface, the user input to modify the bone surface; and
modifying the bone surface according to the user input.

### Claim 5 (depends on 1)

The method of claim 1, wherein the tissue depth dataset is selected from a plurality of tissue depth datasets based on a known characteristic of the live actor.

### Claim 6 (depends on 5)

The method of claim 5, wherein each dataset of the plurality of tissue depth datasets corresponds to any combination of following factors:
race;
age;
gender;
height;
weight; and
body mass index.

### Claim 7

A system for generating a bone surface depicting at least a portion of a skull of a live actor, the system comprising:
an input interface configured to obtain a plurality of facial scans of the live actor, each facial scan including a respective skin surface and respective sensing data indicative of facial muscle strains corresponding to a set of facial muscles;
a communication interface configured to obtain, from a database, a tissue depth dataset including a tissue depth indicating a distance from a corresponding surface point on the human face to a skull surface underneath the human face;
one or more processors configured to:
derive a bone surface based on a skin surface generated from the plurality of facial scans and the tissue depth dataset;
refine the bone surface based on a user input via a user interface; and
refine the bone surface via a neural network by inputting to the neural network the plurality of facial scans and a tissue depth array from the tissue depth dataset, and using muscle or skin parameters derived from the plurality of facial scans as ground truth labels.

### Claim 8 (depends on 7)

The system of claim 7, wherein the one or more processors are further configured to determine, from the plurality of facial scans, the three-dimensional facial skin surface structure of the live actor by:
determining, for a sampled point on the three-dimensional facial skin surface structure, a corresponding point on the bone surface based on a respective tissue depth corresponding to the sampled point;
determining the bone surface by interpolating a plurality of determined corresponding points on the bone surface.

### Claim 9 (depends on 7)

The system of claim 7, wherein the one or more processors are further configured to:
generate a muscle model from the plurality of the facial scans of the live actor, wherein the muscle model comprises a plurality of facial muscles and each facial muscle includes an attachment point to the skull; and
refine the bone surface according to a set of attachment points indicated by the generated muscle model.

### Claim 10 (depends on 7)

The system of claim 7, wherein the one or more processors are further configured to:
receive, via the user interface, the user input to modify the bone surface; and
modify the bone surface according to the user input.

### Claim 11 (depends on 8)

The system of claim 8, wherein the tissue depth dataset is selected from a plurality of tissue depth datasets based on a known characteristic of the live actor.

### Claim 12 (depends on 11)

The system of claim 11, wherein each dataset of the plurality of tissue depth datasets corresponds to any combination of following factors:
race;
age;
gender;
height;
weight; and
body mass index.

### Claim 13

A computer-readable non-transitory medium storing computer-executable instructions for generating a bone surface depicting at least a portion of a skull of a live actor, the computer-executable instructions being executed by one or more processors to perform operations comprising:
obtaining a plurality of facial scans of the live actor, each facial scan including a respective skin surface and respective sensing data indicative of facial muscle strains corresponding to a set of facial muscles;
obtaining, from a database, a tissue depth dataset including a tissue depth indicating a distance from a corresponding surface point on the human face to a skull surface underneath the human face;
deriving a bone surface based on a skin surface generated from the plurality of facial scans and the tissue depth dataset;
refining the bone surface based on a user input via a user interface; and
refining the bone surface via a neural network by inputting to the neural network the plurality of facial scans and a tissue depth array from the tissue depth dataset, and using muscle or skin parameters derived from the plurality of facial scans as ground truth labels.

### Claim 14 (depends on 13)

The computer-readable non-transitory medium of claim 13, further comprising an operation of determining, from the plurality of facial scans, the three-dimensional facial skin surface structure of the live actor that comprises:
determining, for a sampled point on the three-dimensional facial skin surface structure, a corresponding point on the bone surface based on a respective tissue depth corresponding to the sampled point;
determining the bone surface by interpolating a plurality of determined corresponding points on the bone surface.

### Claim 15 (depends on 13)

The computer-readable non-transitory medium of claim 13, wherein the operations further comprise:
generating a muscle model from the plurality of the facial scans of the live actor, wherein the muscle model comprises a plurality of facial muscles and each facial muscle includes an attachment point to the skull; and
refining the bone surface according to a set of attachment points indicated by the generated muscle model.

### Claim 16 (depends on 13)

The computer-readable non-transitory medium of claim 13, wherein the tissue depth dataset is selected from a plurality of tissue depth datasets based on a known characteristic of the live actor.

### Claim 17 (depends on 13)

The computer-readable non-transitory medium of claim 13, wherein each dataset of the plurality of tissue depth datasets corresponds to any combination of following factors:
race;
age;
gender;
height;
weight; and
body mass index.