# US 11,445,266 B2 · System and computerized method for subtitles synchronization of audiovisual content using the human voice detection for synchronization

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,445,266 B2
- Document ID: US11445266B2
- Status: Granted
- Kind: B2
- Application number: 17/200,428
- Filed: 2021-03-12
- Published: 2021-07-01
- Granted: 2022-09-13
- Earliest claimed: 2018-09-13
- Assignees: IChannel.IO Ltd.
- Applicants: IChannel.IO Ltd.
- Inventors: Oren Jack Maurice
- Web page: https://patentgrep.com/patent/US11445266B2

## CPC

- H04N 21/242: Synchronisation processes, e.g. processing of PCR [Programme Clock References] {(arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18)}
- H04N 21/4884: {for displaying subtitles}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/57: for processing of video signals
- G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel
- H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/43074: {of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV programme}
- H04N 21/435: Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
- H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

Audiovisual content in the form of video clip files, streamed or broadcasted may further contain subtitles. Such subtitles are provided with timing information so that each subtitle should be displayed synchronously with the spoken words. However, at times such synchronization with the audio portion of the audiovisual content has a timing offset which when above a predetermined threshold is bothersome. The system and method determine time spans in which a human speaks and attempts to synchronize those time spans with the subtitle content. Indication is provided when an incurable synchronization exists as well as the case where the subtitles and audio are well synchronized. It further is able to determine, when an offset exists, the type of offset (constant or dynamic) and providing the necessary adjustment information so that the timing used in conjunction with the subtitles timing provided may be corrected and synchronization deficiency resolved.

## Claims

### Claim 1

A method for synchronization of subtitles and human voice segments of an audiovisual content comprises:
gathering a plurality of audio segments of the audiovisual content;
gathering a plurality of subtitles of the audiovisual content;
generating a list of possible start offsets and end offsets for the plurality of audio segments and the plurality of subtitles;
determine a best cost and a plurality of factor values by performing a calculation on each of the start offsets and end offsets of the plurality of audio segments and the plurality of subtitles;
generating a notification that no match has been found upon determination that the best cost is not less than a value defined as infinity;
generating a notification that no correction is necessary upon determination that the plurality of factors have each a predetermined value and that the best cost is less than a value defined as infinity; and
performing a subtitle offset correction upon determination that the plurality of factors do not have each a predetermined value and that the best cost is less than a value defined as infinity.

### Claim 2 (depends on 1)

The method of claim 1, wherein the plurality of audio segments were created using a voice activity detection technique.

### Claim 3 (depends on 1)

The method of claim 1, wherein a first factor of the plurality of factors has a value of ‘0’ and a second factor of the plurality of factors has a value of ‘1’.

### Claim 4 (depends on 1)

The method of claim 1, wherein performing a calculation on each of the start offsets and end offsets of the plurality of audio segments and the plurality of subtitles comprises repetition of:
selecting a start offset and an end offset of an audio segment and a subtitle;
calculating a corrected subtitle time per the selected offsets;
calculating the plurality of factors for the selected offsets;
calculating cost for mismatches and if it is over a predetermined threshold value set the cost to infinity;
save the calculated cost and the plurality of factors upon determination that this is the best cost;
until all the start offsets and the end offsets of the audiovisual content and subtitles have been selected.

### Claim 5 (depends on 1)

The method of claim 1, wherein the offset is a linear drift.

### Claim 6 (depends on 1)

The method of claim 1, wherein the offset is a non-linear drift.