Automatic Dubbing: Methods and Apparatuses

A video dubbing system for automatically dubbing a video signal into a different target language than the original audio while keeping vocal characteristics of original speakers in the audio, comprising:

• a speech separation system capable of processing an audio signal in a video signal and splitting it into a speech signal and a background signal;
• an audio segmenter capable of detecting voice and non-voice segments in the audio signal;
• an ASR system associated with at least one language in the audio signal that is capable of outputting words in sequence and their time stamps;
• a casing and punctuation system that is capable of transforming the word sequence from the ASR system into human readable text with casing and punctuation;
• a visual diarization system capable of receiving the video signal and outputting prior probabilities on the number of speakers in the video based on face detection, lips movement detection and face recognition models;
• a speaker diarization system capable of receiving the voice segments in the audio signal and assigning speaker labels to the voice segments and re-segmenting the audio signal according to the speaker labels;
• a MT system with length control capable of outputting isochronic translations of the human readable text associated with the audio signal;
• a speech rate prediction system that is capable of outputting deviations from a normal speech rate;
• a speech placement system that, for each utterance in the translated text, is capable of assigning time stamps to it based on its duration, on/off screen voice information, and surrounding silence and can also change the speaking rate of each utterance;
• a speaker encoder that is capable of extracting speaker feature vectors from audio speech segments;
• a speech emotion classifier that is capable of receiving audio speech segments and outputting speech emotion label information;
• a text emotion classifier that is capable of receiving text for a speech segment and outputting text emotion label information that can be expressed by the content of the text;
• a speaker-adaptive TTS system that is capable of (i) receiving translated text in a target language, an original audio stream, and at least one speaker feature vector corresponding to at least one speaker represented in the original audio stream, and (ii) synthesizing voices in the target language that controllably sound like the at least one speaker in the original audio;
• and
• a signal processing system that is capable of concatenating synthesized audio signals from the TTS system intertwined with silence according to timings provided by the speech placement system, blending background audio with the concatenated synthesized audio signals to generate a translated audio signal and merging the translated audio signal with the original video signal to generate a new dubbed video signal.

A system for speaker-adaptive dubbing in different languages, comprising:

• a memory that stores program instructions for execution to process an audio stream that includes speech in a first language and text in a target language corresponding to the speech;
• an input/output unit that receives video and audio signals and data;
• and
• a processor, coupled to the memory, that is capable of executing the program instructions to (i) receive translated text in a target language, an original audio stream, and at least one speaker feature vector corresponding to at least one speaker represented as speech in the original audio stream, wherein the original audio stream includes speech in a first language different from the target language, and (ii) synthesize voices in the target language based on the original audio stream and the at least one speaker feature vector corresponding to at least one speaker represented as speech in the original audio stream that controllably sounds like the at least one speaker in the original audio.