# US 12,111,880 B2 · Face swapping with neural network-based geometry refining

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,111,880 B2
- Document ID: US12111880B2
- Status: Granted
- Kind: B2
- Application number: 17/484,681
- Filed: 2021-09-24
- Published: 2022-11-24
- Granted: 2024-10-08
- Earliest claimed: 2021-05-20
- Assignees: Disney Enterprises, Inc.; ETH Zurich (Eidgenssische Technische Hochschule Zurich)
- Applicants: Disney Enterprises, Inc.; ETH Zürich (Eidgenössische Technische Hochschule Zürich)
- Inventors: Jacek Krzysztof Naruniec; Derek Edward Bradley; Paulo Fabiano Urnau Gotardo; Leonhard Markus Helminger; Christopher Andreas Otto; Christopher Richard Schroers; Romann Matthew Weber
- Web page: https://patentgrep.com/patent/US12111880B2

## CPC

- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06T 19/20: Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06T 17/20: Finite element generation, e.g. wire-frame surface description, {tesselation}
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/647: {by matching two-dimensional images to three-dimensional objects}
- G06V 40/161: {Detection; Localisation; Normalisation}
- G06V 40/172: {Classification, e.g. identification}
- G06T 2207/20081: Training; Learning
- G06T 2207/30201: Face
- G06T 2219/2021: Shape modification
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Various embodiments set forth systems and techniques for changing a face within an image. The techniques include receiving a first image including a face associated with a first facial identity; generating, via a machine learning model, at least a first texture map and a first position map based on the first image; rendering a second image including a face associated with a second facial identity based on the first texture map and the first position map, wherein the second facial identity is different from the first facial identity.

## Claims

### Claim 1

A computer-implemented method for changing a face within an image, the method comprising:
receiving a first image including a face associated with a first facial identity;
generating, via a machine learning model, at least a first texture map, a second texture map, and a first position map based on the first image, wherein the second texture map represents one or more adjustments to the first texture map; and
rendering a second image including a face associated with a second facial identity based on the first texture map, the first position map, and the second texture map, wherein the second facial identity is different from the first facial identity.

### Claim 2 (depends on 1)

The method of claim 1, wherein the machine learning model comprises an encoder and a plurality of decoders, wherein each decoder included in the plurality of decoders is associated with a different facial identity, and wherein generating the first texture map and the first position map uses a first decoder included in the plurality of decoders that is associated with the second facial identity.

### Claim 3

A computer-implemented method for changing a face within an image, the method comprising:
receiving a first image including a face associated with a first facial identity;
generating, via a machine learning model, wherein the machine learning model comprises a decoder including a plurality of sets of layers, wherein each set of layers included in the plurality of sets of layers is associated with a different facial identity, and wherein generating the first texture map and the first position map uses a first set of layers included in the plurality of sets of layers that is associated with the second facial identity; and
rendering a second image including a face associated with a second facial identity based on the first texture map and the first position map, wherein the second facial identity is different from the first facial identity.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first texture map represents an average texture map corresponding to the second facial identity, and the second texture map represents one or more adjustments to the average texture map.

### Claim 5 (depends on 1)

The method of claim 1, wherein rendering the second image comprises generating a composite texture map based on the first texture map and the second texture map.

### Claim 6 (depends on 1)

The method of claim 1, further comprising training the machine learning model based on a plurality of training input images, wherein the plurality of training input images includes one or more neutral input images associated with the first facial identity, one or more neutral input images associated with the second facial identity, one or more non-neutral input images associated with the first facial identity, and one or more non-neutral input images associated with the second facial identity.

### Claim 7 (depends on 6)

The method of claim 6, wherein training the machine learning model comprises, for each training input image included in the plurality of training input images:
generating, using the machine learning model, training output corresponding to the training input image;
computing one or more of a reconstruction loss, a silhouette loss, or a smoothing loss based on the training output; and
refining the machine learning model based on the one or more of the reconstruction loss, the silhouette loss, or the smoothing loss.

### Claim 8 (depends on 1)

The method of claim 1, wherein the machine learning model comprises a plurality of decoders, wherein each decoder included in the plurality of decoders corresponds to a different facial identity, the method further comprises training each decoder included in the plurality of decoders based on a different set of training images associated with the corresponding facial identity.

### Claim 9 (depends on 1)

The method of claim 1, wherein rendering the second image comprises generating a 3D mesh based on the first position map, wherein rendering the second image is further based on the 3D mesh.

### Claim 10 (depends on 1)

The method of claim 1, further comprising generating a vertex displacement map based on the first image, wherein generating the first texture map and the first position map is further based on the vertex displacement map.

### Claim 11

One or more non-transitory computer-readable media including instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
receiving a first image including a face associated with a first facial identity;
generating, via a machine learning model, at least a first texture map, a second texture map, and a first position map based on the first image, wherein the second texture map comprises one or more adjustments to the first texture map; and
rendering a second image including a face associated with a second facial identity based on the first texture map, the first position map, and the second texture map, wherein the second facial identity is different from the first facial identity.

### Claim 12 (depends on 11)

The non-transitory computer-readable media of claim 11, wherein the machine learning model comprises an encoder and a plurality of decoders, wherein each decoder included in the plurality of decoders is associated with a different facial identity, and wherein generating the first texture map and the first position map uses a first decoder included in the plurality of decoders that is associated with the second facial identity.

### Claim 13 (depends on 11)

The non-transitory computer-readable media of claim 11, wherein the first texture map represents an average texture map corresponding to the second facial identity, and the second texture map represents one or more adjustments to the average texture map.

### Claim 14 (depends on 11)

The non-transitory computer-readable media of claim 11, wherein rendering the second image comprises generating a composite texture map based on the first texture map and the second texture map.

### Claim 15 (depends on 11)

The non-transitory computer-readable media of claim 11, wherein the first position map represents an average geometry corresponding to the second facial identity, wherein the instructions, when executed by the one or more processors further cause the one or more processors to perform the step of generating a second position map that represents one or more adjustments to the average geometry, and wherein rendering the second image is further based on the second position map.

### Claim 16 (depends on 15)

The non-transitory computer-readable media of claim 15, wherein rendering the second image comprises generating a composite position map based on the first position map and the second position map.

### Claim 17 (depends on 16)

The non-transitory computer-readable media of claim 16, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of generating a vertex displacement map based on the first image, and wherein generating the composite position map is further based on the vertex displacement map.

### Claim 18 (depends on 16)

The non-transitory computer-readable media of claim 16, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the steps of:
generating a 3D mesh based on the composite position map; and
modifying the 3D mesh based on the first image.

### Claim 19 (depends on 11)

The non-transitory computer-readable media of claim 11, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to perform the step of generating a vertex displacement map based on the first image, wherein generating the first texture map and the first position map is further based on the vertex displacement map.

### Claim 20

A system comprising:
one or more memories storing instructions;
one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to:
receive a first image including a face associated with a first facial identity;
generate, via a machine learning model, at least a first texture map, a second texture map, and a first position map based on the first image, wherein the second texture map comprises one or more adjustments to the first texture map; and
render a second image including a face associated with a second facial identity based on the first texture map, the first position map, and the second texture map, wherein the second facial identity is different from the first facial identity.