Optimization of lip syncing in natural language translated video

A computer system for optimizing generated lip-synced videos for neural machine translations, the system comprising:

• one or more computer processors;
• one or more computer readable storage media;
• computer program instructions to;
• receive a source video of a speaker in a source language;
• synthesize a plurality of speech translations of the source video in a target language, where the target language is different than the source language, based on a neural machine translation model, wherein the neural machine translation model is trained to synthesize a variable number of translations based on a variable beam width, wherein synthesizing comprises:
• extract audio data from the source video, generate a transcript in the source language for the extracted audio data;
• generate at least one translation script in the target language based on the transcript in the source language, wherein the number of translation scripts is based on the variable beam width;
• and convert each translation script into speech based on a text to speech system, wherein the translation script is the topK result of the translation into the target language;
• generate a lip synchronized video for each of the plurality of synthesized speech translations based on a generation sub-model within a generative adversarial network architecture;
• classify each lip synchronized video as synchronized or not synchronized, based on a classification sub-model within a generative adversarial network;
• and
• generate a lip-sync score for each lip synchronized video classified as synchronized.

A computer program product for optimizing generated lip-synced videos for neural machine translations, the computer program product comprising one or more computer readable storage media and program instructions sorted on the one or more computer readable storage media, the program instructions comprising instructions to:

• receive a source video of a speaker in a source language;
• synthesize a plurality of speech translations of the source video in a target language, where the target language is different than the source language, based on a neural machine translation model, wherein the neural machine translation model is trained to synthesize a variable number of translations based on a variable beam width, wherein synthesizing comprises:
• extract audio data from the source video, generate a transcript in the source language for the extracted audio data;
• generate at least one translation script in the target language based on the transcript in the source language, wherein the number of translation scripts is based on the variable beam width;
• and convert each translation script into speech based on a text to speech system, wherein the translation script is the topK result of the translation into the target language;
• generate a lip synchronized video for each of the plurality of synthesized speech translations based on a generation sub-model within a generative adversarial network architecture;
• classify each lip synchronized video as synchronized or not synchronized, based on a classification sub-model within a generative adversarial network;
• and
• generate a lip-sync score for each lip synchronized video classified as synchronized.