US 2026/0197534 A1Published application

Method and system for enhancing lip synchronization in video generation with new speech

AssigneeConstructor Technology AG

InventorsSergey Ulasen, Andrey Adashchik, Karsten Kozempel, Nwafor Chinedu Kenneth, Serg Bell, Stanislav Protasov, Nikolay Dobrovolskiy, Laurent Dedenis

Life of this patent

Estimating the term…

patentgrep

Loading patent…

Abstract

A method and system is disclosed for generating video content where speech audio is accurately synchronized with facial movements. The method begins by converting input text into speech audio using a text-to-speech (TTS) system. An original video is then processed to identify and extract the mouth area of the human face. The method generates a plurality of frames of the mouth area corresponding to segments of the speech audio using a trained generative neural network. These frames are lip-synced with the corresponding audio segments. The extracted mouth area frames are overlaid onto the original video, followed by performing facial reconstruction using a trained facial reconstruction model to correct distortions and achieve a natural and expressive appearance around the mouth area. Finally, the generated speech audio is combined with the modified video to produce a final video with synchronized lip movements and phonetic sounds.

Claims15

Dependent claims

A computer-implemented method for generating a video for rendering speech, the method comprising:

receiving an original video, wherein the original video comprises a human face rendering a speech;
processing the original video to extract a mouth area of the human face;
generating a speech audio from an input text using a text-to-speech (TTS) system;
generating a plurality of frames of the mouth area corresponding to a plurality of audio segments in the speech audio, wherein the mouth area in each frame is lip-synched with a corresponding audio segment of the plurality of audio segments in the speech audio using a pre-trained generative neural network;
overlaying the plurality of frames of the mouth area over the human face in the original video;
performing facial reconstruction of the human face in each frame of the original video using a trained facial reconstruction model in response to the overlaying, wherein the facial reconstruction corrects distorted and blurry areas around the mouth area to achieve a natural, sharp, and expressive facial representation;
and
adding the speech audio to the original video in response to performing the facial reconstruction for generating the video for rendering speech.

5 more elements

A computer system comprising a processor and one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions executable by the processor to cause the processor to perform operations comprising:

receiving an original video, wherein the original video comprises a human face rendering a speech;
processing the original video to extract a mouth area of the human face;
generating a speech audio from an input text using a text-to-speech (TTS) system;
generating a plurality of frames of the mouth area corresponding to a plurality of audio segments in the speech audio, wherein the mouth area in each frame is lip-synched with a corresponding audio segment of the plurality of audio segments in the speech audio using a pre-trained generative neural network;
overlaying the plurality of frames of the mouth area over the human face in the original video;
performing facial reconstruction of the human face in each frame of the original video using a trained facial reconstruction model in response to the overlaying, wherein the facial reconstruction corrects distorted and blurry areas around the mouth area to achieve a natural, sharp, and expressive facial representation;
and
adding the speech audio to the original video in response to performing the facial reconstruction for generating the video for rendering speech.

5 more elements

Dependencies are inferred from the claim text.