Systems and methods for improved lip dubbing

A computer-generated method for generating output video, the method comprising:

• receiving an initial video data object V comprising a plurality of image frame data objects I;
• extracting, from an initial audio data object A corresponding to initial video data object V, an initial set of expression/viseme codes corresponding to spectrogram segments in the initial audio data object A;
• receiving a set of time-synchronized viseme parameters corresponding to target audio data object A′;
• and
• processing the initial video data object V and the set of time-synchronized viseme parameters using a machine learning network to generate an output video data object V′, wherein initial mouth regions in the initial video data object V have been replaced with replacement mouth regions generated based on the set of time-synchronized viseme parameters;
• wherein the initial video data object V is utilized to populate a viseme library of available visemes based on the initial set of expression/viseme codes;
• and
• wherein the method further comprises comparing visemes identified in the target audio data object A′ to the viseme library of available visemes based on the initial set of expression/viseme codes to identify a subset of visemes not in the viseme library and requiring synthesis.

A non-transitory computer readable medium, storing machine interpretable instruction sets, which, when executed by a processor, cause the processor to perform the steps of:

• receiving an initial video data object V comprising a plurality of image frame data objects I;
• extracting, from an initial audio data object A corresponding to initial video data object V, an initial set of expression/viseme codes corresponding to spectrogram segments in the initial audio data object A;
• receiving a set of time-synchronized viseme parameters corresponding to target audio data object A′;
• and
• processing the initial video data object V and the set of time-synchronized viseme parameters using a machine learning network to generate an output video data object V′, wherein initial mouth regions in the initial video data object V have been replaced with replacement mouth regions generated based on the set of time-synchronized viseme parameters;
• wherein the initial video data object V is utilized to populate a viseme library of available visemes based on the initial set of expression/viseme codes;
• and
• wherein the method further comprises comparing visemes identified in the target audio data object A′ to the viseme library of available visemes based on the initial set of expression/viseme codes to identify a subset of visemes not in the viseme library and requiring synthesis.