# US 12,182,920 B2 · Method and apparatus for generating mouth shape by using deep learning network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,182,920 B2
- Document ID: US12182920B2
- Status: Granted
- Kind: B2
- Application number: 18/570,595
- Filed: 2022-10-25
- Published: 2024-08-08
- Granted: 2024-12-31
- Earliest claimed: 2021-10-25
- Assignees: Klleon Inc.
- Applicants: Klleon Inc.
- Inventors: Ji-Su Kang
- Web page: https://patentgrep.com/patent/US12182920B2

## CPC

- G06V 40/169: {Holistic features and representations, i.e. based on the facial image taken as a whole}
- G06N 3/08: Learning methods
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 11/60: Creating or editing images; Combining images with text
- G06T 13/205: {driven by audio data}
- G06T 13/40: of characters, e.g. humans, animals or virtual beings
- G06V 40/16: Human faces, e.g. facial parts, sketches or expressions
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 21/10: Transforming into visible information
- G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads}
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

The present invention relates to a method for generating a mouth shape by using a deep learning network, and comprises the steps of: receiving a face image as an input; generating a mouth shape of the face image into a preset mouth shape by using a first mouth shape generation preprocessing deep learning model; receiving speech information as an input; generating the preset mouth shape into a mouth shape synchronizing with the speech information by using a second mouth shape generation preprocessing deep learning model; and transforming the mouth shape synchronizing with the speech information into high definition by using a high definition transform postprocessing deep learning model.

## Claims

### Claim 1

A method for generating a mouth shape by using a deep learning network, the method comprising:
(a) a step at which an image input unit receives a face image as an input;
(b) a step at which by using a first mouth shape generation preprocessing deep learning model, a first mouth shape generation unit extracts a facial feature vector for an entire face from the face image, extracts a speech feature vector from data of speech uttered with a mouth closed, and combines the facial feature vector and the speech feature vector to transform a mouth shape of the face image into a mouth shape with the mouth closed;
(c) a step at which a speech input unit receives speech information as an input;
(d) a step at which a second mouth shape generation unit generates the mouth shape with the mouth closed into a mouth shape synchronizing with the speech information by using a second mouth shape generation preprocessing deep learning model; and
(e) a step at which the high-definition transform unit transforms the mouth shape synchronizing with the speech information into a high-definition mouth shape by using a high-definition transform postprocessing deep learning model;
wherein the step (d) comprises:
(d1) a step of extracting a facial feature vector for an entire face from a face image having the mouth shape when the mouth is closed;
(d2) a step of extracting a speech feature vector from the speech information; and
(d3) a step of combining the facial feature vector and the speech feature vector;
wherein the second mouth shape generation preprocessing deep learning model is a deep learning model that beforehand has learned the generation of the mouth shape synchronizing with the speech information on a basis of a loss function in a learning stage;
wherein the loss function is to calculate a similarity between a first multiplication operation value and a second multiplication value;
wherein the first multiplication operation value is a result of a face image generated by the second mouth shape generation preprocessing deep learning model multiplying a mouth mask of the mouth by a Hadamard product;
wherein the second multiplication operation value is a result of the input face image multiplying the mouth mask by a Hadamard product;
wherein the loss function is expressed as:
Mask Loss=∥Generated Face⊙Mouth Mask,Original Face⊙Mouth Mask∥1;
wherein Generated Face refers to the face image generated by the second mouth shape generation preprocessing deep learning model, ⊙ refers to a Hadamard product, Mouth Mask refers to the mouth mask, and Original Face refers to the input face image.

### Claim 2 (depends on 1)

The method of claim 1, wherein the step (e) comprises:
(e1) a step of extracting a facial feature vector for an entire face from a high-definition face image of the face image input from the step (a);
(e2) a step of extracting a facial feature vector for an entire face from a face image having the mouth shape synchronizing with the speech information; and
(e3) a step of combining the extracted facial feature vectors.

### Claim 3

An apparatus for generating a mouth shape by using a deep learning network, the apparatus comprising:
an image input unit configured to receive a face image as an input;
a first mouth shape generation unit configured to extract a facial feature vector for an entire face from the face image, to extract a speech feature vector from data of speech uttered with a mouth closed, and to combine the facial feature vector and the speech feature vector to transform a mouth shape of the face image into a mouth shape with the mouth closed by using a first mouth shape generation preprocessing deep learning model;
a speech input unit configured to receive speech information as an input;
a second mouth shape generation unit configured to generate the mouth shape with the mouth closed into a mouth shape synchronizing with the speech information by using a second mouth shape generation preprocessing deep learning model; and
a high-definition transform unit configured to transform the mouth shape synchronizing with the speech information into a high-definition mouth shape by using a high-definition transform postprocessing deep learning model;
wherein the second mouth shape generation unit extracts a facial feature vector for an entire face from a face image having the mouth shape when the mouth is closed,
extracts a speech feature vector from the speech information, and
combines the facial feature vector and the speech feature vector;
wherein the second mouth shape generation preprocessing deep learning model is a deep learning model that beforehand has learned the generation of the mouth shape synchronizing with the speech information on a basis of a loss function in a learning stage;
wherein the loss function is to calculate a similarity between a first multiplication operation value and a second multiplication value;
wherein the first multiplication operation value is a result of a face image generated by the second mouth shape generation preprocessing deep learning model multiplying a mouth mask of the mouth by a Hadamard product;
wherein the second multiplication operation value is a result of the input face image multiplying the mouth mask by a Hadamard product;
wherein the loss function is expressed as:
Mask Loss=∥Generated Face⊙Mouth Mask,Original Face⊙Mouth Mask∥1;
wherein Generated Face refers to the face image generated by the second mouth shape generation preprocessing deep learning model, ⊙ refers to a Hadamard product, Mouth Mask refers to the mouth mask, and Original Face refers to the input face image.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the high-definition transform unit extracts a facial feature vector for an entire face from a high-definition face image of the face image received by the image input unit,
extracts a facial feature vector for an entire face from a face image having the mouth shape synchronizing with the speech information, and
combines the extracted facial feature vectors.