Audio-speech driven animated talking face generation using a cascaded generative adversarial network

A system, comprising:

• a memory storing instructions;
• one or more communication interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
• obtain an audio speech and a set of identity images (SI) of a target individual;
• extracting one or more DeepSpeech features of the target individual from the audio speech;
• generate, using the extracted DeepSpeech features, via a first generative adversarial network (FGAN) of a cascaded GAN comprised in the memory (102), a speech-induced motion (SIM) on a sparse representation (SR) of a neutral mean face, wherein the SR of the SIM comprises a plurality of facial landmark points with one or more finer deformations of lips;
• generate a plurality of eye blink movements from random noise input learnt from a video dataset, wherein the plurality of eye blink movements are generated for each eye based on a sequence of generated displacements of associated facial landmark points of each eye region, and wherein the plurality of eye blink movements comprise a set of eye landmark points with blink motion;
• replace one or more eye landmark points of the plurality of facial landmark points with the set of eye landmark points with blink motion to obtain a set of final landmark points, the set of final landmark points comprises (i) the set of eye landmark points with blink motion and (ii) one or more landmark points containing the speech-induced motion;
• generate one or more target-specific landmark points (TSLP) based on (i) the set of final landmark points (FLP) and (ii) an identity landmark obtained from the set of identity images;
• determine at least one face type as one of a pre-stored face image or a new face image using the set of identity images;
• and
• perform, via a second generative adversarial network (SGAN) of the cascaded GAN executed by the one or more hardware processors, based on the at least one determined face type, one of:
• fine-tuning a meta-trained network (MN) using the set of identity images (SI) and the plurality of facial landmark points extracted from the set of identity images (SI) for the target individual to obtain a fine-tuned meta-trained network (FTMTN) and generating, using the one or more target-specific landmark points (TSLP), a meta-learning-based texture (MLT) via the FTMTN thereof;
• or
• generating a meta-learning-based texture (MLT) using the set of identity images (SI) and the one or more target-specific landmark points (TSLP), via a pre-generated fine-tuned meta-trained network (FTMTN), wherein the generated MLT serves as an animated talking face of the target individual.

A computer program product comprising a non-transitory computer readable medium having a computer readable program embodied therein, wherein the computer readable program, when executed on a computing device causes the computing device to generate audio-speech driven animated talking face using a cascaded generative adversarial network by

• obtaining, via one or more hardware processors, an audio speech and a set of identity images of a target individual;
• extracting, via the one or more hardware processors, one or more DeepSpeech features of the target individual from the audio speech;
• generating, using the extracted DeepSpeech features, via a first generative adversarial network (FGAN) of the cascaded GAN executed by the one or more hardware processors, a speech-induced motion (SIM) on a sparse representation (SR) of a neutral mean face, wherein the SR of the SIM comprises a plurality of facial landmark points with one or more finer deformations of lips;
• generating, via the one or more hardware processors, a plurality of eye blink movements from random noise input learnt from a video dataset, wherein the plurality of eye blink movements are generated for each eye based on a sequence of generated displacements of associated facial landmark points of each eye region, and wherein the plurality of eye blink movements comprise a set of eye landmark points with blink motion;
• replacing one or more eye landmark points of the plurality of facial landmark points with the set of eye landmark points with blink motion to obtain a set of final landmark points, the set of final landmark points comprises (i) the set of eye landmark points with blink motion and (ii) one or more landmark points containing the speech-induced motion;
• generating, via the one or more hardware processors one or more target-specific landmark points (TSLP) based on (i) the set of final landmark points (FLP) and (ii) an identity landmark obtained from the set of identity images;
• determining at least one face type as one of a pre-stored face image or a new face image using the set of identity images;
• and
• performing, via a second generative adversarial network (SGAN) of the cascaded GAN executed by the one or more hardware processors, based on the at least one determined face type, one of:
• fine-tuning a meta-trained network (MN) using the set of identity images (SI) and the plurality of facial landmark points extracted from the SI for the target individual to obtain a fine-tuned meta-trained network (FTMTN) and generating, using the one or more target-specific landmark points (TSLP), a meta-learning texture (MLT) via the FTMTN thereof;
• or
• generating a meta-learning texture (MLT) using the SI and the one or more target-specific landmark points (TSLP), via a pre-generated fine-tuned meta-trained network (FTMTN), the generated MLT serves as an animated talking face of the target individual.