Text and audio-based real-time face reenactment

A system for text and audio-based real-time face reenactment, the system comprising at least one processor, a memory storing processor-executable codes, wherein the at least one processor is configured to implement the following operations upon executing the processor-executable codes:

• receiving an input text and a target image, the target image including a target face;
• generating, based on the input text, a sequence of sets of acoustic features representing the input text;
• generating, based on the sequence of sets of acoustic features, a sequence of sets of scenario data, the sets of scenario data indicating modifications of facial key points of a model face pronouncing the input text, wherein the
• generating the sequence of sets of scenario data includes:
• generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points of the model face by providing the sets of acoustic features to a first model, the first model being trained to generate, based on a set of acoustic features, a set of mouth key points, the first model being trained based on first images of a first actor pronouncing first training input texts;
• and
• generating, based on the sequence of sets of mouth key points, a sequence of sets of the facial key points of the model face by providing the sets of mouth key points to a second model, the second model being trained to generate, based on the set of mouth key points, a set of the facial key points, the second model being trained based on second images of a second actor pronouncing second training input texts;
• generating, based on the sequence of sets of scenario data and the target image, a sequence of frames, wherein each of the frames includes the target face modified based on at least one set of scenario data of the sequence of sets of scenario data;
• and
• generating, based on the sequence of frames, an output video.

A non-transitory processor-readable medium having instructions stored thereon, which when executed by one or more processors, cause the one or more processors to implement a method for text and audio- based real-time face reenactment, the method comprising:

• receiving an input text and a target image, the target image including a target face;
• generating, based on the input text, a sequence of sets of acoustic features representing the input text;
• generating, based on the sequence of sets of acoustic features, a sequence of sets of scenario data, the sets of scenario data indicating modifications of facial key points of a model face pronouncing the input text, wherein the generating the sequence of sets of scenario data includes:
• generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points of the model face by providing the sets of acoustic features to a first model, the first model being trained to generate, based on a set of acoustic features, a set of mouth key points, the first model being trained based on first images of a first actor pronouncing first training input texts;
• and
• generating, based on the sequence of sets of mouth key points, a sequence of sets of the facial key points of the model face by providing the sets of mouth key points to a second model, the second model being trained to generate, based on the set of mouth key points, a set of the facial key points, the second model being trained based on second images of a second actor pronouncing second training input texts;
• generating, based on the sequence of sets of scenario data and the target image, a sequence of frames, wherein each of the frames includes the target face modified based on at least one set of scenario data of the sequence of sets of scenario data;
• and
• generating, based on the sequence of frames, an output video.