An apparatus for generating a mouth shape by using a deep learning network, the apparatus comprising:
- an image input unit configured to receive a face image as an input;
- a first mouth shape generation unit configured to extract a facial feature vector for an entire face from the face image, to extract a speech feature vector from data of speech uttered with a mouth closed, and to combine the facial feature vector and the speech feature vector to transform a mouth shape of the face image into a mouth shape with the mouth closed by using a first mouth shape generation preprocessing deep learning model;
- a speech input unit configured to receive speech information as an input;
- a second mouth shape generation unit configured to generate the mouth shape with the mouth closed into a mouth shape synchronizing with the speech information by using a second mouth shape generation preprocessing deep learning model;
- and
- a high-definition transform unit configured to transform the mouth shape synchronizing with the speech information into a high-definition mouth shape by using a high-definition transform postprocessing deep learning model;
- wherein the second mouth shape generation unit extracts a facial feature vector for an entire face from a face image having the mouth shape when the mouth is closed,
- extracts a speech feature vector from the speech information, and
- combines the facial feature vector and the speech feature vector;
- wherein the second mouth shape generation preprocessing deep learning model is a deep learning model that beforehand has learned the generation of the mouth shape synchronizing with the speech information on a basis of a loss function in a learning stage;
- wherein the loss function is to calculate a similarity between a first multiplication operation value and a second multiplication value;
- wherein the first multiplication operation value is a result of a face image generated by the second mouth shape generation preprocessing deep learning model multiplying a mouth mask of the mouth by a Hadamard product;
- wherein the second multiplication operation value is a result of the input face image multiplying the mouth mask by a Hadamard product;
- wherein the loss function is expressed as:
- Mask Loss=∥Generated Face⊙Mouth Mask,Original Face⊙Mouth Mask∥1;
- wherein Generated Face refers to the face image generated by the second mouth shape generation preprocessing deep learning model, ⊙ refers to a Hadamard product, Mouth Mask refers to the mouth mask, and Original Face refers to the input face image.