Audio and video translator
AssigneeDeep Media Inc.
InventorsRijul Gupta, Emma Brown
Life of this patent
Estimating the term…
Abstract
A system and method for translating audio, and video when desired. The translations include synthetic media and data generated using AI systems. Through unique processors and generators executing a unique sequence of steps, the system and method produces more accurate translations that can account for various speech characteristics (e.g., emotion, pacing, idioms, sarcasm, jokes, tone, phonemes, etc.). These speech characteristics are identified in the input media and synthetically incorporated into the translated outputs to mirror the characteristics in the input media. Some embodiments further include systems and methods that manipulate the input video such that the speakers' faces and/or lips appear as if they are natively speaking the generated audio.
Claims20
A system configured to generate a synthetic media with translated speech corresponding to an input media file, comprising:
- at least one processor;
- and
- memory including instructions that, when executed by the at least one processor, cause the system to:
- digitally acquire the input media file, wherein the input media file includes input audio in a first input language;
- acquire an input transcription, wherein the input transcription includes text corresponding to the words spoken in the input audio;
- acquire input meta information, the meta information including emotion data, wherein emotion data corresponds to one or more detectable emotions from a list of predetermined emotions;
- input the input meta information into a meta translation generator;
- and
- translate the first input language into a first output language that is distinct from the first input language, wherein translating is based at least on the emotion data.
A method for generating a synthetic media with translated speech corresponding to an input media file, comprising:
- acquiring the input media file, wherein the input media file includes input video and input audio in a first input language;
- acquiring an input transcription, wherein the input transcription includes text corresponding to the words spoken in the input audio;
- acquire input meta information, the meta information including tone data;
- translate the input transcription and input meta information into a first output language that is distinct from the first input language;
- and
- generate translated audio by providing the translated input transcription and meta information to an audio translation generator configured to generate the translated audio.
A non-transitory computer readable medium for generating a synthetic media with translated speech corresponding to an input media file, comprising instructions stored thereon, that when executed on at least one processor, cause the at least one processor to:
- digitally acquire the input media file, wherein the input media file includes input audio in a first input language;
- acquire an input transcription, wherein the input transcription includes text corresponding to the words spoken in the input audio;
- acquire input meta information, the meta information including emotion data;
- input the input meta information and input transcription into a transcription and meta translation generator;
- and
- translate, by the transcription and meta translation generator, the input transcription and input meta information into a first output language that is distinct from the first input language, wherein translating is based at least on the emotion data.
Dependencies are inferred from the claim text.