Back to results
US 10,930,263 B1Granted
Automatic voice dubbing for media content localization
AssigneeAmazon Technologies, Inc.
InventorHooman Mahyar
Life of this patent
Estimating the term…
Abstract
This disclosure describes techniques for replicating characteristics of an actor or actresses voice across different languages. The disclosed techniques have the practical application of enabling automatic generation of dubbed video content for multiple languages, with particular speakers in each dubbing having the same voice characteristics as the corresponding speakers in the original version of the video content.
Claims20
A method, including:
- receiving, by a first voice synthesis model trained using deep learning to replicate a first set of speech features for a first actor, a first vector, the first vector including a phonetic transcription of first subtitle text for video content, the first vector also including an audio characteristic indicating a manner of speech associated with the first subtitle text;
- generating, using the first voice synthesis model, a first predicted audio waveform corresponding to a prediction of the first subtitle text being spoken in the indicated manner of speech in a first language by the first actor in the video content;
- receiving, by a second voice synthesis model trained using reinforcement learning to replicate a second set of speech features for the first actor, the first vector;
- generating, using the second voice synthesis model, a second predicted audio waveform corresponding to a prediction of the first subtitle text being spoken in the indicated manner of speech in the first language by the first actor;
- receiving, by a voice synthesis model selector for the first actor, the first predicted audio waveform, the second predicted audio waveform, and a third audio waveform, the third audio waveform representing the first subtitle text being spoken by the first actor in a second language that is different than the first language;
- determining, by the voice synthesis model selector, a first similarity in a duration of the first predicted audio waveform and a duration of the third audio waveform, and a second similarity in a duration of the second predicted audio waveform and the duration of the third audio waveform;
- selecting, by the voice synthesis model selector, the first audio waveform based in part on determining that the first similarity between the first predicted audio waveform and the third audio waveform indicates more similarity in duration than the second similarity between the second predicted audio waveform and the third audio waveform;
- receiving, by a voice tuner, the first predicted audio waveform selected by the voice synthesis model selector;
- generating, by the voice tuner, a fourth audio waveform by modifying at least one audio localization parameter of the first predicted audio waveform, the fourth audio waveform representing a third prediction of the first subtitle text being spoken in the first language by the first actor;
- and
- providing, to a video dubbing server, data for associating playback of the fourth audio waveform with playback of image frames of the video content corresponding to the first subtitle text.
8 more elements
A method, including:
- receiving, by a first neural network trained to represent a first set of speech features for a first speaker, a first vector, the first vector including a representation of first subtitle text for video content, the first vector also including an audio characteristic associated with the first subtitle text;
- generating, using the first neural network, a first predicted audio of the first subtitle text being spoken in a first language by the first speaker in the video content;
- receiving, by a second neural network trained to represent a second set of speech features for the first speaker, the first vector;
- generating, using the second neural network, a second predicted audio of the first subtitle text being spoken in the first language by the first speaker;
- receiving, by a predicted audio selector for the first speaker, the first predicted audio, the second predicted audio, and third audio, the third audio representing the first subtitle text being spoken by the first speaker in a second language that is different than the first language;
- determining, by the predicted audio selector, a first similarity between the first predicted audio and the third audio, and a second similarity between the second predicted audio and the third audio;
- selecting, by the predicted audio selector, the first audio based in part on determining that the first similarity between the first predicted audio and the third audio indicates more similarity than the second similarity between the second predicted audio and the third audio;
- receiving, by a voice tuner, the first predicted audio selected by the predicted audio selector;
- generating, by the voice tuner, fourth audio by modifying at least one audio localization parameter of the first predicted audio, the fourth audio representing a third prediction of the first subtitle text being spoken in the first language by the first speaker;
- and
- providing, to a video dubbing server, data for associating playback of the fourth audio with playback of image frames of the video content corresponding to the first subtitle text.
8 more elements
A system, comprising one or more computing devices configured to:
- receive, by a first neural network trained to represent a first set of speech features for a first speaker, a first vector, the first vector including a representation of first subtitle text for video content, the first vector also including an audio characteristic associated with the first subtitle text;
- generate, using the first neural network, a first predicted audio of the first subtitle text being spoken in a first language by the first speaker in the video content;
- receive, by a second neural network trained to represent a second set of speech features for the first speaker, the first vector;
- generate, using the second neural network, a second predicted audio of the first subtitle text being spoken in the first language by the first speaker;
- receive, by a predicted audio selector for the first speaker, the first predicted audio, the second predicted audio, and third audio, the third audio representing the first subtitle text being spoken by the first speaker in a second language that is different than the first language;
- determine, by the predicted audio selector, a first similarity between the first predicted audio and the third audio, and a second similarity between the second predicted audio and the third audio;
- select, by the predicted audio selector, the first audio based in part on determining that the first similarity between the first predicted audio and the third audio indicates more similarity than the second similarity between the second predicted audio and the third audio;
- receive, by a voice tuner, the first predicted audio selected by the predicted audio selector;
- generate, by the voice tuner, fourth audio by modifying at least one audio localization parameter of the first predicted audio, the fourth audio representing a third prediction of the first subtitle text being spoken in the first language by the first speaker;
- and
- provide, to a video dubbing server, data for associating playback of the fourth audio with playback of image frames of the video content corresponding to the first subtitle text.
8 more elements
Dependencies are inferred from the claim text.