Method and system for generating 2D animated lip images synchronizing to an audio signal

A lip generator system, to generate 2D animated lip images synchronizing to an audio signal of an unseen target subject, comprising:

• a memory storing instructions;
• one or more communication interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
• receive, by a lip generator system, an audio signal and a target lip image of an unseen target subject as inputs from a user, wherein the lip generator system is trained using a meta learning based model agnostic technique, wherein the lip generator system comprises a meta trained and finetuned audio encoder, a meta trained and finetuned image encoder and a meta trained and finetuned decoder, wherein each 2D animated lip image is synchronized to the input audio signal, and train the lip generator system using the meta learning based model agnostic technique to:
• perform meta learning on a training dataset comprising a plurality of data, wherein each data of the plurality of data comprises a subject's lip image and an audio signal as input and the corresponding subject's sequence of lip images synchronized to the input audio signal as output, wherein each training dataset is split into a plurality of episodes;
• iteratively, perform for every episode of the plurality of episodes:
• (i) split, each episode into a predefined number of tasks among a plurality of tasks, wherein each task comprises a predefined number of training data samples and a predefined number of test data samples corresponding to one subject;
• (ii) initialize one or more fast weights and one or more global weights, wherein the one or more features corresponding to each individual subject are learnt using the one or more fast weights, wherein the one or more global weights are updated based on the fast weights for global representation of the one or more features for the training dataset corresponding to all the subjects;
• (iii) iteratively perform for every task from the plurality of tasks of the current episode, comprises:
• a) copying, the one or more global weights into the one t weights;
• b) updating, the one or more fast weights using a gradient descent technique for training data samples corresponding to the selected task among the plurality of tasks;
• and
• c) determining, the loss on the test data samples corresponding to the subject of the task based on the one or more fast weights;
• and
• (iv) updating, the one or more global weights based on the loss accumulated based on the plurality of tasks in the current episode;
• extract, by the meta trained and finetuned audio encoder, a plurality of high dimensional audio features from the input audio signal, and extract, by the meta trained and finetuned image encoder, a plurality of high dimensional image features from the target lip image;
• generate, a plurality of high dimensional audio image features based on by fusing the plurality of high dimensional audio features with the plurality of high dimensional image features;
• and
• generate, by the meta trained and finetuned decoder, a plurality of 2D animated lip images of the unseen target subject to synchronize with the input audio signal based on the plurality of high dimensional audio image features serving as an input to the meta trained and finetuned decoder.

One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors perform actions comprising:

• receive, by a lip generator system, an audio signal and a target lip image of an unseen target subject as inputs from a user, wherein the lip generator system is trained using a meta learning based model agnostic technique, wherein the lip generator system comprises a meta trained and finetuned audio encoder, a meta trained and finetuned image encoder and a meta trained and finetuned decoder, wherein each 2D animated lip image is synchronized to the input audio signal, wherein training the lip generator system using the meta learning based model agnostic technique further comprises:
• performing meta learning on a training dataset comprising a plurality of data, wherein each data of the plurality of data comprises a subject's lip image and an audio signal as input and the corresponding subject's sequence of lip images synchronized to the input audio signal as output, wherein each training dataset is split into a plurality of episodes;
• iteratively, performing for every episode of the plurality of episodes:
• (i) splitting, each episode into a predefined number of tasks among a plurality of tasks, wherein each task comprises a predefined number of training data samples and a predefined number of test data samples corresponding to one subject;
• (ii) initializing one or more fast weights and one or more global weights, wherein the one or more features corresponding to each individual subject are learnt using the one or more fast weights, wherein the one or more global weights are updated based on the fast weights for global representation of the one or more features for the training dataset corresponding to all the subjects;
• (iii) iteratively performing for every task from the plurality of tasks of the current episode, comprises:
• a) copying, the one or more global weights into the one or more fast weights;
• b) updating, the one or more fast weights using a gradient descent technique for training data samples corresponding to the selected task among the plurality of tasks;
• and
• c) determining, the loss on the test data samples corresponding to the subject of the task based on the one or more fast weights;
• and
• (iv) updating, the one or more global weights based on the loss accumulated based on the plurality of tasks in the current episode;
• extract by the meta trained and finetuned audio encoder a plurality of high dimensional audio features from the input audio signal, and extracting, by the meta trained and finetuned image encoder, a plurality of high dimensional image features from the target lip image;
• generate a plurality of high dimensional audio image features based on the plurality of high dimensional audio features with the plurality of high dimensional image features;
• and
• generate, by the meta trained and finetuned decoder, a plurality of 2D animated lip images of the unseen target subject to synchronize with the input audio signal based on the plurality of high dimensional audio image features serving as an input to the meta trained and finetuned decoder.