# US 2022/0284916 A1 · Determining subtitle synchronization when compared to audio track

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284916 A1
- Document ID: US20220284916A1
- Status: Published application
- Kind: A1
- Application number: 17/450,548
- Filed: 2021-10-11
- Published: 2022-09-08
- Earliest claimed: 2021-03-05
- Assignees: NBCUniversal Media, LLC
- Applicants: Nbcuniversal Media, LLC
- Inventors: Craig Gardner; Christopher Burdorf
- Web page: https://patentgrep.com/patent/US20220284916A1

## CPC

- G10L 21/055: for synchronising with other signals, e.g. video signals
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 25/57: for processing of video signals
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel
- H04N 5/278: Subtitling
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

According to at least one embodiment, a method for determining a level of synchronicity between subtitle text in audiovisual content and speech that would be heard during display of the audiovisual content includes: accessing a first binary sequence, each bit of the first binary sequence indicating whether the speech is provided at a respective sampling time of a plurality of sampling times; and accessing a second binary sequence, each bit of the second binary sequence indicating whether the subtitle text is provided at a respective sampling time of a plurality of sampling times. The method further includes comparing the first binary sequence and the second binary sequence to determine the level of synchronicity between the subtitle text and the speech.

## Claims

### Claim 1

A method for determining a level of synchronicity between subtitle text in audiovisual content and speech that would be heard during display of the audiovisual content, the method comprising:
accessing a first binary sequence, each bit of the first binary sequence indicating whether the speech is provided at a respective sampling time of a plurality of sampling times;
accessing a second binary sequence, each bit of the second binary sequence indicating whether the subtitle text is provided at a respective sampling time of a plurality of sampling times; and
comparing the first binary sequence and the second binary sequence to determine the level of synchronicity between the subtitle text and the speech.

### Claim 2 (depends on 1)

The method of claim 1, wherein accessing the first binary sequence comprises performing an audio analysis of an audio track corresponding to the audiovisual content, to determine whether the speech is provided at each of the plurality of sampling times.

### Claim 3 (depends on 2)

The method of claim 2, wherein performing the audio analysis of the audio track comprises performing a voice detection to determine whether the audio track includes audio of a human voice at each of the plurality of sampling times.

### Claim 4 (depends on 2)

The method of claim 2, further comprising deriving the plurality of sampling times based on an initial time code and a frame rate of the audio track.

### Claim 5 (depends on 1)

The method of claim 1, wherein accessing the second binary sequence comprises determining whether a subtitle file provides the subtitle text at each of the plurality of sampling times.

### Claim 6 (depends on 1)

The method of claim 1, wherein adjacent sampling times of the plurality of sampling times are spaced apart by a periodic interval.

### Claim 7 (depends on 1)

The method of claim 1, wherein comparing the first binary sequence and the second binary sequence comprises performing a cross-correlation between the first binary sequence and the second binary sequence.

### Claim 8 (depends on 1)

The method of claim 1, wherein comparing the first binary sequence and the second binary sequence is based on an average cross-correlation value that is based on a respective cross-correlation value between the first binary sequence and the second binary sequence at each of a plurality of sample intervals

### Claim 9 (depends on 1)

The method of claim 1, wherein comparing the first binary sequence and the second binary sequence comprises analyzing a phase difference between the first binary sequence and the second binary sequence.

### Claim 10 (depends on 1)

The method of claim 1, wherein comparing the first binary sequence and the second binary sequence is based on an average phase difference that is based on a respective phase difference between the first binary sequence and the second binary sequence at each of a plurality of sample intervals.

### Claim 11

An apparatus for determining a level of synchronicity between subtitle text in audiovisual content and speech that would be heard during display of the audiovisual content, the apparatus comprising:
a network communication unit configured to transmit and receive data; and
one or more controllers configured to:
access a first binary sequence, each bit of the first binary sequence indicating whether the speech is provided at a respective sampling time of a plurality of sampling times;
access a second binary sequence, each bit of the second binary sequence indicating whether the subtitle text is provided at a respective sampling time of a plurality of sampling times; and
compare the first binary sequence and the second binary sequence to determine the level of synchronicity between the subtitle text and the speech.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the one or more controllers are further configured to access the first binary sequence by performing an audio analysis of an audio track corresponding to the audiovisual content, to determine whether the speech is provided at each of the plurality of sampling times.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the one or more controllers are further configured to perform the audio analysis of the audio track by performing a voice detection to determine whether the audio track includes audio of a human voice at each of the plurality of sampling times.

### Claim 14 (depends on 12)

The apparatus of claim 12, wherein the one or more controllers are further configured to derive the plurality of sampling times based on an initial time code and a frame rate of the audio track.

### Claim 15 (depends on 11)

The apparatus of claim 11, wherein the one or more controllers are further configured to access the second binary sequence by determining whether a subtitle file provides the subtitle text at each of the plurality of sampling times.

### Claim 16 (depends on 11)

The apparatus of claim 11, wherein the one or more controllers are further configured to compare the first binary sequence and the second binary sequence by performing a cross-correlation between the first binary sequence and the second binary sequence.

### Claim 17 (depends on 11)

The apparatus of claim 11, wherein comparing the first binary sequence and the second binary sequence is based on an average cross-correlation value that is based on a respective cross-correlation value between the first binary sequence and the second binary sequence at each of a plurality of sample intervals.

### Claim 18 (depends on 11)

The apparatus of claim 11, wherein the one or more controllers are further configured to compare the first binary sequence and the second binary sequence by analyzing a phase difference between the first binary sequence and the second binary sequence.

### Claim 19 (depends on 11)

The apparatus of claim 11, wherein comparing the first binary sequence and the second binary sequence is based on an average phase difference that is based on a respective phase difference between the first binary sequence and the second binary sequence at each of a plurality of sample intervals.

### Claim 20

A machine-readable non-transitory medium having stored thereon machine-executable instructions for determining a level of synchronicity between subtitle text in audiovisual content and speech that would be heard during display of the audiovisual content, the instructions comprising:
accessing a first binary sequence, each bit of the first binary sequence indicating whether the speech is provided at a respective sampling time of a plurality of sampling times;
accessing a second binary sequence, each bit of the second binary sequence indicating whether the subtitle text is provided at a respective sampling time of a plurality of sampling times; and
comparing the first binary sequence and the second binary sequence to determine the level of synchronicity between the subtitle text and the speech.
