# US 11,908,449 B2 · Audio and video translator

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,449 B2
- Document ID: US11908449B2
- Status: Granted
- Kind: B2
- Application number: 18/070,626
- Filed: 2022-11-29
- Published: 2023-03-23
- Granted: 2024-02-20
- Earliest claimed: 2021-05-05
- Assignees: Deep Media Inc.
- Applicants: Deep Media Inc.
- Inventors: Rijul Gupta; Emma Brown
- Web page: https://patentgrep.com/patent/US11908449B2

## CPC

- G06F 40/30: Semantic analysis
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G06F 40/103: Formatting, i.e. changing of presentation of documents (automatic justification G06F40/189; automatic line break hyphenation G06F40/191)
- G06F 40/42: Data-driven translation
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06V 10/778: Active pattern-learning, e.g. online learning of image or video features
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G10L 15/04: Segmentation; Word boundary detection
- G10L 17/00: Speaker identification or verification techniques
- G06T 2207/30201: Face
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system and method for translating audio, and video when desired. The translations include synthetic media and data generated using AI systems. Through unique processors and generators executing a unique sequence of steps, the system and method produces more accurate translations that can account for various speech characteristics (e.g., emotion, pacing, idioms, sarcasm, jokes, tone, phonemes, etc.). These speech characteristics are identified in the input media and synthetically incorporated into the translated outputs to mirror the characteristics in the input media. Some embodiments further include systems and methods that manipulate the input video such that the speakers' faces and/or lips appear as if they are natively speaking the generated audio.

## Claims

### Claim 1

A system configured to generate a synthetic media with translated speech corresponding to an input media file, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the system to:
digitally acquire the input media file, wherein the input media file includes input audio in a first input language;
segment the input audio into one or more vocal segments, wherein each vocal segment includes a speaker identification to identify the speaker of each vocal segment;
for each vocal segment:
acquire an input transcription, wherein the input transcription includes text corresponding to the words spoken in each vocal segment;
acquire input meta information, the meta information including emotion data and tone data, wherein emotion data corresponds to one or more detectable emotions from a list of predetermined emotions;
input the input meta information and input transcription into a transcription and meta translation generator, wherein the transcription and meta translation generator is a generative adversarial network generator; and
translate, by the transcription and meta translation generator, the input transcription and input meta information into a first output language that is distinct from the first input language, wherein translating is based at least on the emotion data.

### Claim 2 (depends on 1)

The system of claim 1, wherein the input media file is in a computer-readable format.

### Claim 3 (depends on 1)

The system of claim 1, wherein the instructions included in the memory further cause the system to partition one vocal stream from another, reduce background noise, or enhance a quality of the vocal streams.

### Claim 4 (depends on 1)

The system of claim 1, wherein the instructions included in the memory further cause the system to capture lip movement tracking data from the input media file.

### Claim 5 (depends on 1)

The system of claim 1, wherein the instructions included in the memory further cause the system to identify pacing information for each word or phoneme in each vocal segment.

### Claim 6 (depends on 5)

The system of claim 5, wherein segmenting the input audio and identifying pacing information is performed by a speaker diarization processor configured to receive the input media file.

### Claim 7 (depends on 1)

The system of claim 1, wherein the text of the input transcription is formatted according to the international phonetics alphabet.

### Claim 8 (depends on 1)

The system of claim 1, wherein the input transcription further includes sentiment analysis and tracking data corresponding to anatomical landmarks for the speaker for each vocal segment.

### Claim 9 (depends on 1)

The system of claim 1, wherein acquiring the input transcription of the input audio includes providing the input audio to an artificial intelligence (AI) generator configured to convert the input audio into text.

### Claim 10 (depends on 1)

The system of claim 1, wherein acquiring input meta information includes providing the input audio and the input transcription to an AI meta information processor configured to identify meta information.

### Claim 11 (depends on 1)

The system of claim 1, wherein the instructions included in the memory further cause the system to provide the translated transcription and meta information to the audio translation generator which is configured to generate translated audio.

### Claim 12 (depends on 11)

The system of claim 11, wherein the instructions included in the memory further cause the system to stitch the translated audio for each vocal segment back into a single audio file.

### Claim 13 (depends on 11)

The system of claim 11, wherein the input media file includes input video and the instructions included in the memory further cause the system to provide translated audio and the input video to a video sync generator and generating, by the video sync generator, a synced video in which the translated audio syncs with the input video.

### Claim 14

A non-transitory computer-readable medium for generating a synthetic media with translated speech corresponding to an input media file, comprising instructions stored thereon, that when executed on at least one processor, cause the at least one processor to:
acquire the input media file, wherein the input media file includes input video and input audio in a first input language;
segment the input audio into one or more vocal segments, wherein each vocal segment includes a speaker identification to identify the speaker of each vocal segment;
for each vocal segment:
acquire an input transcription, wherein the input transcription includes text corresponding to the words spoken in each vocal segment;
acquire input meta information, the meta information including emotion data and tone data, wherein emotion data corresponds to one or more detectable emotions from a list of predetermined emotions;
translate the input transcription and input meta information into a first output language that is distinct from the first input language; and
generate translated audio by providing the translated input transcription and meta information to an audio translation generator configured to generate the translated audio, wherein the audio translation generator is a generative adversarial network generator.

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the input media file is in a computer-readable format.

### Claim 16 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the instructions further cause the at least one processor to capture lip movement tracking data from the input video.

### Claim 17 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the input transcription is formatted according to the international phonetics alphabet.

### Claim 18 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the instructions further cause the at least one processor to identify pacing information for each word or phoneme in each vocal segment.

### Claim 19 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the instructions further cause the at least one processor to stitch the translated audio for each vocal segment back into a single audio file.

### Claim 20 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the instructions further cause the at least one processor to:
provide the translated audio and the input video to a video sync generator; and
generate, by the video sync generator, a synced video in which the translated audio syncs with the input video to create the synthetic media.