System and method for an audio-visual avatar creation

A system for generating an audio-visual avatar specific to an application, the system comprising:

• a processor;
• and instructions that, when executed by the processor, cause the processor to implement:
• a general synthesizer to receive an audio training dataset and a video training dataset comprising:
• a voice synthesis module trained by the audio training dataset to divide the audio training dataset corresponding to a plurality of audio phenomena, each audio phenomena comprising at least one pattern of an audio phrase with characteristic expression, audio phonetic symbol, or audio-dependent time code, wherein the audio training dataset comprises audio samples not of the target person and wherein the voice synthesis module is trained for text-to-speech comprising at least one user-modified element added to voice synthesis module embeddings,
• a video synthesis module trained by the video training dataset to divide the video training dataset into a plurality of frames corresponding to a plurality of video phenomena, each video phenomena comprising at least one pattern of a video phrase with characteristic expression, video phonetic symbol, or video-dependent time code, wherein the video training dataset comprises video samples not of the target person,
• a customized synthesizer, characterizing a target person, to receive an audio sample and a video sample of the target person, wherein the video sample of the target person has been previously recorded according to a predefined video script and predefined video requirements, the customized synthesizer comprising:
• a voice custom synthesis module trained on the audio sample of the target person,
• a video custom synthesis module trained on the video sample of the target person by training a first neural network using the video sample of the target person to segment and classify body movements of the target person into a plurality of sequences corresponding to a phenomenon of the target person including fractionated torso movements characteristic of the target person for the phenomenon and fractionated head movements characteristic of the target person for the phenomenon,
• a video generator to create an audio-visual avatar including by:
• receive, by a voice cloning system, an input text to be converted into an audio clip,
• synthesize a voice clone, by voice cloning system, from the input text the voice synthesis module and the voice custom synthesis module,
• process, by the voice cloning system, the voice clone to be formatted by phrases and expressions,
• synthesize a video clone, by a video cloning system, based on the video synthesis module and the video custom synthesis module including:
• retrieving a body movement script comprising a predefined description for recognizing body movement sequences with associated speech,
• extracting body movement parameters from the first neural network based on the body movement script, the body movement parameters characteristic of the target person for the phenomenon,
• retrieving a face movement script comprising a predefined description for recognizing face movement sequences with associated speech,
• extracting face movement parameters from the first neural network based on the body movement script, the face movement parameters characteristic of the target person for the phenomenon,
• wherein based on recorded gestures from the video sample of the target person and a target gesture script, a preliminary video of the head of the target person is generated with controlled gestures but without lip synchronization, the target gesture script comprising a predefined description for recognizing target gesture sequences with associated speech, and
• wherein the preliminary video is compared with a voice recording of the target person to match a plurality of lip movements with words spoken by the target person in the preliminary video to perform lip synchronization,
• apply the voice clone to the video clone for creating the audio-visual avatar by the video generator, the video clone including the body movement parameters characteristic of the target person for the phenomenon and the face movement parameters characteristic of the target person for the phenomenon.