# US 12,114,048 B2 · Automated voice translation dubbing for prerecorded videos

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,114,048 B2
- Document ID: US12114048B2
- Status: Granted
- Kind: B2
- Application number: 18/109,243
- Filed: 2023-02-13
- Published: 2023-06-22
- Granted: 2024-10-08
- Earliest claimed: 2023-02-13
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Terrance Paul McCartney, Jr.; Brian Colonna; Michael Nechyba
- Web page: https://patentgrep.com/patent/US12114048B2

## CPC

- H04N 21/4884: {for displaying subtitles}
- G06F 40/30: Semantic analysis
- G06F 40/58: Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- H04N 21/2335: {involving reformatting operations of audio signals, e.g. by converting from one coding standard to another (details of audio signal transcoding G10L19/173)}
- H04N 21/2343: involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements {(video transcoding H04N19/40; media packet handling at the source H04L65/762)}
- H04N 21/234336: {by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text}
- H04N 21/235: Processing of additional data, e.g. scrambling of additional data or processing content descriptors
- H04N 21/43074: {of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV programme}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method for aligning a translation of original caption data with an audio portion of a video is provided. The method involves identifying original caption data for the video that includes caption character strings, identifying translated language caption data for the video that includes translated character strings associated with audio portion of the video, and mapping caption sentence fragments generated from the caption character strings to corresponding translated sentence fragments generated from the translated character strings based on timing associated with the original caption data and the translated language caption data. The method further involves estimating time intervals for individual caption sentence fragments using timing information corresponding to individual caption character strings, assigning time intervals to individual translated sentence fragments based on estimated time intervals of the individual caption sentence fragments, generating a set of translated sentences using consecutive translated sentence fragments, and aligning the set of translated sentences with the audio portion of the video using assigned time intervals of individual translated sentence fragments from corresponding translated sentences.

## Claims

### Claim 1

A method comprising:
identifying, by a processing device, original caption data for a video having audio that includes speech recorded in an original language, wherein the original caption data comprises a plurality of caption character strings in the original language and associated with audio portion of the video;
identifying, by the processing device, translated language caption data for the video, wherein the translated language caption data comprises a plurality of translated character strings associated with the audio portion of the video;
generating, by the processing device, a set of caption sentence fragments from the plurality of caption character strings and a set of translated sentence fragments from the plurality of translated character strings;
mapping, by the processing device, caption sentence fragments of the set of caption sentence fragments to corresponding translated sentence fragments of the set of translated sentence fragments based on timing associated with the original caption data and the translated language caption data;
estimating, by the processing device, time intervals for individual caption sentence fragments of the set of caption sentence fragments using timing information corresponding to individual caption character strings;
assigning, by the processing device, time intervals to individual translated sentence fragments of the set of translated sentence fragments based on estimated time intervals of the individual caption sentence fragments;
generating, by the processing device, a set of translated sentences using consecutive translated sentence fragments of the set of translated sentence fragments; and
aligning, by the processing device, the set of translated sentences with the audio portion of the video using assigned time intervals of individual translated sentence fragments from corresponding translated sentences.

### Claim 2 (depends on 1)

The method of claim 1 wherein the set of caption sentence fragments is generated from the plurality of caption character strings is generated using punctuation in the plurality of caption character strings.

### Claim 3 (depends on 1)

The method of claim 1 further comprising:
identifying speech recognition data generated for the audio portion of the video, the speech recognition data comprising a plurality of generated character strings and associated timing information for each generated character string.

### Claim 4 (depends on 3)

The method of claim 3 further comprising determining the timing associated with the original caption data by mapping the plurality of caption character strings to the plurality of generated character strings, and assigning timing information to individual caption character strings of the original caption data based on timing information of mapped individual generated character strings, wherein the timing associated with the original caption data corresponds to the timing information assigned to the individual caption character strings of the original caption data.

### Claim 5 (depends on 4)

The method of claim 4 wherein the plurality of caption character strings to the plurality of generated character strings are mapped using assigned values indicative of semantic similarities between the individual caption character strings and individual generated character strings of the plurality of generated character strings.

### Claim 6 (depends on 4)

The method of claim 4 further comprising prior to mapping the plurality of caption character strings to the plurality of generated character strings, normalizing the plurality of caption character strings and the plurality of generated character strings by removing non-spoken character strings.

### Claim 7 (depends on 5)

The method of claim 5 wherein mapping the plurality of caption character strings to the plurality of generated character strings using the assigned values indicative of semantic similarities between the individual caption character strings of the plurality of caption character strings and the individual generated character strings of the plurality of generated character strings comprises:
assigning an integer identifier value to each unique character string of the individual caption character strings of the plurality of caption character strings and each unique character string of the individual generated character strings of the plurality of generated character strings;
wherein a distance between a first integer identifier value associated with a first character string and a second integer identifier value associated with a second character string represents character differences between the first character string and the second character string.

### Claim 8 (depends on 5)

The method of claim 5, wherein mapping the plurality of caption character strings to the plurality of generated character strings using the assigned values comprises:
determining matching pairs of sequences of caption character strings in the plurality of caption character strings and sequences of generated character strings in the plurality of generated character strings using the assigned values of each individual caption character string of the plurality of caption character strings and each individual caption character string of the plurality of generated character strings; and
mapping the individual caption character strings of the plurality of caption character strings to the individual generated character strings of the plurality of generated character strings based on the matching pairs of the sequences of caption character strings and the sequences of generated character strings.

### Claim 9 (depends on 1)

The method of claim 1, generating the set of translated sentences using consecutive translated sentence fragments of the set of translated sentence fragments comprises:
generating a set of caption sentences using consecutive caption sentence fragments of the set of caption sentence fragments based on punctuation; and
generating the set of translated sentences based on the set of caption sentences using machine translation.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
identifying speech segment data for the video, wherein the speech segment data comprises a plurality of speech segments with speech timing information associated with the audio portion of the video; and
updating the time intervals for the individual caption sentence fragments of the set of caption sentence fragments based on speech timing information associated with the plurality of speech segments.

### Claim 11 (depends on 1)

The method of claim 1, further comprising:
obtaining translated audio speech that is machine-generated speech for translation of the original caption data, wherein the translated audio speech comprises a set of translated audio speech segments;
overlaying translated audio speech segments of the set of translated audio speech segments onto video segments of the video that correspond to time intervals for each of the translated audio speech segments; and
generating a second video containing a video portion of the video and a translated audio portion comprising the overlaid translated audio speech segments.

### Claim 12 (depends on 11)

The method of claim 11, wherein overlaying the translated audio speech segments of the set of translated audio speech segments onto the video segments of the video comprises:
for each translated audio speech segment of the set of translated audio speech segments:
determining whether timing information of the translated audio speech segment matches the timing information of a corresponding video segment;
upon determining the timing information of the translated audio speech segment does not match the timing information of the corresponding video segment, adjusting playback speed rate of the translated audio speech segment to match the timing information the translated audio speech segment to the timing information of the corresponding video segment; and
overlaying the translated audio speech segment onto the corresponding video segment.

### Claim 13 (depends on 11)

The method of claim 11, wherein overlaying the translated audio speech segments of the set of translated audio speech segments onto the video segments of the video comprises:
for each translated audio speech segment of the set of translated audio speech segments:
determining whether timing information of the translated audio speech segment matches the timing information of a corresponding video segment;
upon determining the timing information of the translated audio speech segment does not match the timing information of the corresponding video segment, adjusting playback speed rate of the corresponding video segment to match the timing information of the corresponding video segment to the timing information of the translated audio speech segment; and
overlaying the translated audio speech segment onto the corresponding video segment.

### Claim 14

A non-transitory machine-readable storage medium storing instructions which, when executed, cause a processing device to perform operations comprising:
identifying original caption data for a video having audio that includes speech recorded in an original language, wherein the original caption data comprises a plurality of caption character strings in the original language and associated with audio portion of the video;
identifying translated language caption data for the video, wherein the translated language caption data comprises a plurality of translated character strings associated with the audio portion of the video;
generating a set of caption sentence fragments from the plurality of caption character strings and a set of translated sentence fragments from the plurality of translated character strings;
mapping caption sentence fragments of the set of caption sentence fragments to corresponding translated sentence fragments of the set of translated sentence fragments based on timing associated with the original caption data and the translated language caption data;
estimating time intervals for individual caption sentence fragments of the set of caption sentence fragments using timing information corresponding to individual caption character strings;
assigning time intervals to individual translated sentence fragments of the set of translated sentence fragments based on estimated time intervals of the individual caption sentence fragments;
generating a set of translated sentences using consecutive translated sentence fragments of the set of translated sentence fragments; and
aligning the set of translated sentences with the audio portion of the video using assigned time intervals of individual translated sentence fragments from corresponding translated sentences.

### Claim 15 (depends on 14)

The non-transitory machine-readable storage medium of claim 14 wherein the set of caption sentence fragments is generated from the plurality of caption character strings is generated using punctuation in the plurality of caption character strings.

### Claim 16 (depends on 14)

The non-transitory machine-readable storage medium of claim 14, the operations further comprising:
identifying speech recognition data generated for the audio portion of the video, the speech recognition data comprising a plurality of generated character strings and associated timing information for each generated character string.

### Claim 17 (depends on 16)

The non-transitory machine-readable storage medium of claim 16, the operations further comprising determining the timing associated with the original caption data by mapping the plurality of caption character strings to the plurality of generated character strings, and assigning timing information to individual caption character strings of the original caption data based on timing information of mapped individual generated character strings, wherein the timing associated with the original caption data corresponds to the timing information assigned to the individual caption character strings of the original caption data.

### Claim 18 (depends on 17)

The non-transitory machine-readable storage medium of claim 17 wherein the plurality of caption character strings to the plurality of generated character strings are mapped using assigned values indicative of semantic similarities between the individual caption character strings and individual generated character strings of the plurality of generated character strings.

### Claim 19 (depends on 17)

The non-transitory machine-readable storage medium of claim 17, the operations further comprising prior to mapping the plurality of caption character strings to the plurality of generated character strings, normalizing the plurality of caption character strings and the plurality of generated character strings by removing non-spoken character strings.

### Claim 20

A system comprising:
a memory; and
a processing device operatively coupled with the memory to perform operations comprising:
identifying original caption data for a video having audio that includes speech recorded in an original language, wherein the original caption data comprises a plurality of caption character strings in the original language and associated with audio portion of the video;
identifying translated language caption data for the video, wherein the translated language caption data comprises a plurality of translated character strings associated with the audio portion of the video;
generating a set of caption sentence fragments from the plurality of caption character strings and a set of translated sentence fragments from the plurality of translated character strings;
mapping caption sentence fragments of the set of caption sentence fragments to corresponding translated sentence fragments of the set of translated sentence fragments based on timing associated with the original caption data and the translated language caption data;
estimating time intervals for individual caption sentence fragments of the set of caption sentence fragments using timing information corresponding to individual caption character strings;
assigning time intervals to individual translated sentence fragments of the set of translated sentence fragments based on estimated time intervals of the individual caption sentence fragments;
generating a set of translated sentences using consecutive translated sentence fragments of the set of translated sentence fragments; and
aligning the set of translated sentences with the audio portion of the video using assigned time intervals of individual translated sentence fragments from corresponding translated sentences.