# US 2021/0334595 A1 · Methods and systems for image and voice processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334595 A1
- Document ID: US20210334595A1
- Status: Published application
- Kind: A1
- Application number: 17/249,637
- Filed: 2021-03-08
- Published: 2021-10-28
- Earliest claimed: 2019-08-19
- Assignees: Neon Evolution Inc.
- Applicants: Neon Evolution Inc.
- Inventors: Cody Gustave Berlin; Carl Davis Bogan, III; Kenneth Michael Lande; Jacob Myles Laser; Brian Sung Lee; Anders Øland
- Web page: https://patentgrep.com/patent/US20210334595A1

## CPC

- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06T 7/32: using correlation-based methods
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06T 13/80: Two-dimensional [2D] animation, e.g. using sprites
- G06T 17/00: Three-dimensional [3D] modelling for computer graphics
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 40/168: {Feature extraction; Face representation}
- G06V 40/174: {Facial expression recognition}
- G06T 2207/20081: Training; Learning
- G10L 21/0356: for synchronising with other signals, e.g. video signals
- G10L 2021/0135: {Voice conversion or morphing}
- G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Systems and methods are disclosed configured to train an autoencoder using images that include faces, wherein the autoencoder comprises an input layer, an encoder configured to output a latent image from a corresponding input image, and a decoder configured to attempt to reconstruct the input image from the latent image. An image sequence of a face exhibiting a plurality of facial expressions and transitions between facial expressions is generated and accessed. Images of the plurality of facial expressions and transitions between facial expressions are captured from a plurality of different angles and using different lighting. An autoencoder is trained using source images that include the face with different facial expressions captured at different angles with different lighting, and using destination images that include a destination face. The trained autoencoder is used to generate an output where the likeness of the face in the destination images is swapped with the likeness of the source face, while preserving expressions of the destination face.

## Claims

### Claim 1

(canceled)

### Claim 2

An electronic image processing system, comprising:
a network interface;
at least one computing device;
computer readable memory including instructions operable to be executed by the at least one computing device to perform a set of actions, configuring the at least one computing device to:
access captured images of a plurality of facial expressions of a first CGI face and transitions between facial expressions from a plurality of different angles;
train at least one autoencoder using the captured images that include the first CGI face with different facial expressions captured at different angles, and using destination images that include a first real face, wherein the at least one autoencoder comprises:
an input layer;
an encoder;
wherein the encoder is configured to output a latent image from a corresponding input image;
a decoder configured to attempt to reconstruct the input image from the latent image; and
use the trained autoencoder to generate an output using destination images as an input where the first real face likeness in the destination images is swapped with the likeness of the first CGI face, while preserving expressions of the first real face.

### Claim 3 (depends on 2)

The electronic image processing system as defined in claim 2, wherein the at least one autoencoder includes one or more intermediate networks configured to enhance abstract representation learning and wherein the trained autoencoder output has a HD, 4K UHD, 4K, 8K, or 16K resolution.

### Claim 4 (depends on 2)

The electronic image processing system as defined in claim 2, wherein the electronic image processing system is configured to perform operations comprising:
determining an alignment difference between at least one source image in a training set of source images and at least one destination image; and
based at least on the determined alignment difference determining if an additional image is to be obtained for the training set of source images.

### Claim 5 (depends on 2)

The electronic image processing system as defined in claim 2, wherein the electronic image processing system is configured to perform operations comprising:
determining an alignment difference between at least one source image in a training set of source images and at least one destination image; and
based at least on the determined alignment difference determining if at least one source image is to be deleted from the training set of source images.

### Claim 6 (depends on 2)

The electronic image processing system as defined in claim 2,
wherein the first CGI face is generated based on a physical face captured using a neural network, photogrammetry, laser scanning, and/or light detection and ranging.

### Claim 7 (depends on 2)

The electronic image processing system as defined in claim 2, further comprising:
an array of graphics processing units and/or an array of central processing units; and
a task allocator configured to allocate pre-training tasks, training tasks, and face swapping tasks among the graphics processing units in the array of graphics processing units and/or among the central processing units in the array of central processing units.

### Claim 8 (depends on 2)

The electronic image processing system as defined in claim 2, wherein the electronic image processing system is configured to extract destination images from a video, locate given faces within the extracted destination images, align the located given faces, and store timing data associated with when in the video a given face was identified.

### Claim 9 (depends on 2)

The electronic image processing system as defined in claim 2, wherein the electronic image processing system is configured to provide a user interface comprising a training set of controls and an output video creation set of controls, wherein:
the training set of controls includes a control enabling selection of a destination data set, a control enabling specification of a source data set, a control enabling specification of a training batch size and a control initiating training of the autoencoder.

### Claim 10 (depends on 2)

The electronic image processing system as defined in claim 2, wherein the electronic image processing system is configured to provide a user interface comprising a set of controls enabling a user to:
specify a hidden layer width for the at least one autoencoder,
specify a quantity of hidden layers for the at least one autoencoder,
specify whether padding is to be utilized,
specify a convolutional kernel size,
specify whether the at least one autoencoder is to utilize an intermediate neural network,
specify whether and/or which type of normalization is to be used, and
specify whether a learn mask is to be used.

### Claim 11 (depends on 2)

The electronic image processing system as defined in claim 2, wherein the electronic image processing system is configured to provide a user interface comprising:
an original destination facial image column and a corresponding latent destination facial image column configured to display corresponding facial images generated by the at least one autoencoder encoder,
an original source facial image column and a corresponding latent source facial image column configured to display corresponding facial images generated by the at least one autoencoder encoder,
an output facial image column configured to display a face-swapped image from the at least one autoencoder,
network loss values, and
masks.

### Claim 12

A computer-implemented method comprising:
under control of a hardware computing device configured with specific computer-executable instructions:
training an autoencoder using:
a training source data set source images comprising facial images of a CGI source face from a plurality of different angles and/or with a variety of different lighting characteristics, the CGI source facial images exhibiting a plurality of facial expressions,
wherein the autoencoder comprises:
an input layer;
an encoder, wherein the encoder is configured to output a latent image from a corresponding input image;
a decoder configured to attempt to reconstruct the input image from the latent image; and
using the trained autoencoder to generate an output using destination images,
where a face likeness in the destination images is swapped with the CGI source face likeness, while preserving expressions of the destination face.

### Claim 13 (depends on 12)

The computer-implemented method as defined in claim 12, wherein the autoencoder includes one or more intermediate networks configured to enhance abstract representation learning and wherein the trained autoencoder output has a HD, 4K UHD, 4K, 8K, or 16K resolution.

### Claim 14 (depends on 12)

The computer-implemented method as defined in claim 12, the method further comprising:
determining an alignment difference between at least one source image in at least one training set of source images and at least one destination image; and
based at least on the determined alignment difference, determining if an additional image is to be obtained for the at least one training set of source images.

### Claim 15 (depends on 12)

The computer-implemented method as defined in claim 12, the method further comprising:
determining an alignment difference between at least one source image from at least one training set of source images and at least one destination image; and
based at least on the determined alignment difference, determining if at least one source image is to be deleted from the at least one training set of source images.

### Claim 16 (depends on 12)

The computer-implemented method as defined in claim 12,
wherein the training source data set comprising facial images of the source face is generated based at least in part on a physical face captured using a neural network, photogrammetry, laser scanning, and/or light detection and ranging.

### Claim 17 (depends on 12)

The computer-implemented method as defined in claim 12, wherein training the autoencoder further comprises:
training the encoder using both the training source data set images and the destination images and training the decoder using the training source data set images.

### Claim 18 (depends on 12)

The computer-implemented method as defined in claim 12, the method further comprising:
allocating training tasks and face swapping tasks among graphics processing units in an array of graphics processing units and/or among central processing units in an array of central processing units.

### Claim 19 (depends on 12)

The computer-implemented method as defined in claim 12, further comprising:
enabling a voice track associated with the destination images to be synchronized with the source face in the output using the preserved expressions of the destination face.

### Claim 20 (depends on 12)

The computer-implemented method as defined in claim 12, the method further comprising:
extracting the destination images from a video;
locating instances of the destination face within the extracted destination images; and
aligning the located instances of the destination faces.

### Claim 21 (depends on 12)

The computer-implemented method as defined in claim 12, the method further comprising:
providing a user interface comprising a training set of controls and an output video creation set of controls, wherein:
the training set of controls includes a control enabling selection of a destination data set, a control enabling specification of a source data set, a control enabling specification of a training batch size and a control initiating training of the autoencoder; and
the output video creation set of controls a model selection control enabling selection of a trained autoencoder, a control enabling a destination data set to be selected, a control enabling a source data set to be selected, a control enabling an output file format to be specified, and control initiating creation of an output video.

### Claim 22 (depends on 12)

The computer-implemented method as defined in claim 12, the method further comprising:
providing a user interface comprising a set of controls enabling a user to specify, for the autoencoder:
a hidden layer width,
a quantity of hidden layers,
whether padding is to be utilized,
a convolutional kernel size,
whether the autoencoder is to utilize an intermediate neural network,
whether and/or which type of normalization is to be used, and
whether a learn mask is to be used.

### Claim 23 (depends on 12)

The computer-implemented method as defined in claim 12, the method further comprising:
providing a user interface comprising:
an original destination facial image column and a corresponding latent destination facial image column configured to display corresponding facial images generated by the autoencoder encoder,
an original source facial image column and a corresponding latent source facial image column configured to display corresponding facial images generated by the autoencoder encoder,
an output facial image column configured to display a face-swapped image from the autoencoder,
network loss values, and
masks.

### Claim 24 (depends on 12)

The computer-implemented method as defined in claim 12, the method further comprising:
pre-training the autoencoder using a plurality of images of a plurality of faces prior to training the autoencoder.
