Method and device for generating speech moving image

A method for generating a speech moving image that is executed by a computing device including one or more processors and a memory storing one or more programs executed by the one or more processors, the method comprising:

• receiving, in a first encoder, a person background image in which a portion related to speech of a person that is a video part of the speech moving image of the person is covered with a mask to extract an image feature vector from the person background image and compress the extracted image feature vector;
• receiving, in a second encoder, a speech audio signal that is an audio part of the speech moving image to extract a voice feature vector from the speech audio signal, and compress the extracted voice feature vector;
• generating, in a combination unit, a combination vector by combining the compressed image feature vector output from the first encoder and the compressed voice feature vector output from the second encoder;
• and
• reconstructing, in an image reconstruction unit, the speech moving image of the person with the combination vector as an input,
• wherein the first encoder includes a first feature extraction unit that extracts the image feature vector from the person background image and a first compression unit that compresses the extracted image feature vector, and
• calculating, in the first compression unit, a representative value of an image feature vector for each channel based on the extracted image feature vector,
• wherein the representative value is a mean value of the image feature vector for each channel,
• calculating, in the first compression unit, an image representative feature matrix using the representative value of the image feature vector for each channel as each matrix element, and
• controlling, in the first compression unit, a compressed size of the image feature vector by connecting a fully connected neural network to the image representative feature matrix,
• calculating, in the first compression unit, the mean value of the image feature vector for each channel through Equation 1 below:
• f
• c
• =
• 1
• H
• ⁢
• W
• ⁢
• ∑
• i
• =
• 1
• H
• ∑
• j
• =
• 1
• W
• F
• i
• ,
• j
• ,
• c
• [
• Equation
• ⁢
• 1
• ]
• where fc:
• Mean value of an image feature vector of a c-th channel;
• H:
• Height of the image feature vector;
• W:
• Width of the image feature vector;
• and
• Fi,j,c:
• Image feature vector value of the c-th channel at (i, j) coordinates.