# US 2006/0078305 A1 · Method and apparatus to synchronize audio and video

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2006/0078305 A1
- Document ID: US20060078305A1
- Status: Published application
- Kind: A1
- Application number: 11/124,092
- Filed: 2005-05-09
- Published: 2006-04-13
- Earliest claimed: 2004-10-12
- Assignees: Samsung Electronics Co., Ltd.
- Inventors: Manish Arora; Larycguin Serguei
- Web page: https://patentgrep.com/patent/US20060078305A1

## CPC

- G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B 27/28: by using information signals recorded by the same method as the main recording {(G11B27/22 takes precedence)}
- H04N 5/04: Synchronising (for television systems using pulse code modulation H04N7/56)
- H04N 7/08: Systems for the simultaneous or sequential transmission of more than one television signal, e.g. additional information signals, the signals occupying wholly or partially the same frequency band {, e.g. by time division (H04N7/007 takes precedence)}
- H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 21/422: Input-only peripherals {, i.e. input devices connected to specially adapted client devices}, e.g. global positioning system [GPS] {(input devices also receiving signals from specially adapted client devices H04N21/4104)}
- H04N 21/4305: {Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets (arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083)}
- H04N 21/43072: {of multiple content streams on the same device}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- H04N 5/147: {Scene change detection}
- H04N 5/602: {for digital sound signals}
- H04N 21/426: Internal components of the client {; Characteristics thereof} (H04N5/44 takes precedence)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method and an apparatus to synchronize an audio signal and a video signal. The method includes: displaying video on a screen that corresponds to an audio signal including a high frequency component having a predetermined pattern inserted therein to indicate when a scene change occurs in a video signal, detecting a scene change in the displayed video and detecting the high frequency component having the predetermined pattern in the audio signal, calculating a time difference between a time when the scene change is detected in the displayed video and a time when the high frequency component having the predetermined pattern is detected in the audio signal, and controlling delay times of the audio signal and the video signal according to the calculated time difference. Accordingly, the audio signal and the video signal can be automatically synchronized without performing a separate measuring operation.

## Claims

### Claim 1

A method of synchronizing audio and video signals, the method comprising:
displaying video on a screen including an audio signal including a high frequency component having a predetermined pattern inserted therein to indicate when a scene change occurs in a corresponding video signal;
detecting a scene change in the displayed video and detecting the high frequency component having the predetermined pattern in the audio signal;
calculating a time difference between a time when the scene change is detected in the displayed video and a time when the high frequency component having the predetermined pattern is detected in the audio signal; and
controlling delay times of the audio signal and the video signal according to the calculated time difference.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
inserting re-synchronization information into the audio signal before the operation of displaying the video on the screen.

### Claim 3 (depends on 2)

The method of claim 2, wherein the inserting of the re-synchronization information into the audio signal comprises:
detecting a luminance level per pixel of each frame of the video signal;
calculating a luminance level difference between frames according to the detected luminance levels;
detecting the scene change if the calculated luminance level difference is greater than a predetermined threshold value; and
inserting the high frequency component into the audio signal at a time when the calculated luminance level difference is greater than the predetermined threshold.

### Claim 4 (depends on 3)

The method of claim 3, wherein the calculating of the luminance level difference between frames comprises:
calculating a difference between a luminance level of a current frame and a luminance level of a previous frame for each pixel according to the luminance level per pixel of each frame; and
calculating the luminance level difference between the current and previous frames by accumulating the calculated differences for each pixel.

### Claim 5 (depends on 1)

The method of claim 1, wherein the high frequency component having the predetermined pattern is a high frequency component that is beyond an audible frequency band.

### Claim 6 (depends on 1)

The method of claim 1, wherein the detecting of the scene change comprises:
detecting a luminance level of each frame of the displayed video;
calculating a luminance level difference between frames according to the detected luminance levels; and
detecting the scene change if the calculated luminance level difference is greater than a predetermined threshold value.

### Claim 7

A method of maintaining synchronization in video data, the method comprising:
synchronizing an audio signal with a video signal by inserting a synchronization mark into the audio signal when a scene changes in the video signal;
performing one or more signal processing operations on the video signal and the audio signal that introduce a relative delay between the video signal and the audio signal, respectively; and
re-synchronizing the video signal and the audio signal to account for the relative delay by calculating a time difference between when the scene changes and when the synchronization mark is detected in the audio signal and delaying an output audio signal and an output video signal according to the time difference.

### Claim 8 (depends on 7)

The method of claim 7, wherein the synchronization mark comprises an inaudible high frequency component of more than 20,000 Hertz.

### Claim 9

A method of resynchronizing video data, the method comprising:
receiving unsynchronized video data including a video signal and an audio signal having a relative delay therebetween, and the audio signal having one or more synchronization marks inserted therein to indicate when a scene changes in the video signal; and
synchronizing the audio signal and the video signal to account for the relative delay by calculating a time difference between when the scene changes and when the synchronization mark is detected in the audio signal and delaying an output audio signal and an output video signal according to the time difference.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
displaying the video signal and detecting a scene change using a light sensor by comparing luminance levels between consecutive frames of the displayed video signal; and
detecting the one or more synchronization marks in the audio signal.

### Claim 11

A method of inserting synchronization information into video data, the method comprising:
detecting when a scene changes in a video signal of the video data; and
inserting a synchronization mark into an audio signal of the video data when the scene changes in the video signal to synchronize the video signal and the audio signal for output.

### Claim 12 (depends on 11)

The method of claim 11, wherein the detecting of when the scene changes comprises:
determining a luminance level for each pixel in a current frame;
determining a difference between the luminance level of each pixel in the current frame and a luminance level of each pixel in a previous frame;
accumulating each luminance level difference between the current frame and the previous frame; and
determining whether the accumulated difference exceeds a predetermined threshold.

### Claim 13

A method of synchronizing audio and video signals, comprising:
determining a luminance level difference between current and previous frames of a video signal including an audio signal;
adding an identifier to the corresponding audio signal whenever the luminance level difference in the video signal is determined to be greater than a predetermined threshold value; and
controlling delay of output of the audio signal and the video signal based on the added identifier and luminance level difference of the video signal determined to be greater than the predetermined threshold value.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
performing post processing operations on the delayed audio and video signals; and
re-delaying output of the post processed audio and video signals based on the added identifier and luminance level difference of the video signal determined to be greater than the predetermined threshold value.

### Claim 15

An audio and video synchronization apparatus, comprising:
a first scene change detector to calculate a luminance level difference between frames of a video signal and to detect whether the luminance level difference is greater than a predetermined threshold value;
a high frequency inserter to insert a high frequency component having a predetermined pattern into an audio signal whenever the first scene change detector detects that the luminance level difference is greater than the predetermined threshold value;
a signal detector to calculate a luminance level difference between frames from a screen on which the video is displayed, to detect whether the luminance level difference is greater than a predetermined threshold value, and to detect the inserted high frequency component in the audio signal;
a sync difference calculator to calculate a time difference between a time when the signal detector detects that the luminance level difference is greater than the predetermined threshold value and a time when the high frequency component is detected by the signal detector; and
a delay unit to control buffering of the audio signal and the video signal according to the time difference calculated by the sync difference calculator.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the first scene change detector calculates a difference between a luminance level of a current frame and a luminance level of a previous frame for each pixel by detecting a luminance level per pixel of each frame of the video signal and calculates the luminance level difference between the current and previous frames by accumulating the calculated differences of each pixel.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein the signal detector comprises:
a light sensor to sense a luminance level of each frame from a screen on which the video is displayed;
a second scene change detector to calculate a luminance level difference between frames according to the sensed luminance levels and to detect whether the calculated luminance level difference is greater than a predetermined threshold value; and
a high frequency detector to detect the inserted high frequency component from the audio signal.

### Claim 18 (depends on 15)

The apparatus of claim 15, further comprising:
a post processing unit to perform a first one or more processing operations on the audio signal and a second one or more processing operations on the video signal after the high frequency component is inserted into the audio signal to indicate the scene change, and the first and the second one or more processing operations causing a first delay and a second delay in the post processed audio signal and the post processed video signal, respectively.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the time difference calculated by the sync difference calculator equals a difference between the first delay and the second delay.

### Claim 20 (depends on 18)

The apparatus of claim 18, wherein:
the first one or more processing operations include at least one of 3D audio processing of the audio signal, virtualizing the audio signal, enhancing bass of the audio signal, enhancing dialog and clarity of the audio signal, and changing a format of the audio signal; and
the second one or more processing operations include at least one of enhancing color of the video signal, enhancing details of the video signal, enhancing contrast and brightness of the video signal, and changing a format of the video signal.

### Claim 21 (depends on 15)

The apparatus of claim 15, wherein the delay unit comprises:
a variable audio buffer to delay outputting the audio signal according to the time difference calculated by the sync calculator; and
a variable video buffer to delay outputting the video signal according to the time difference calculated by the sync calculator.

### Claim 22 (depends on 15)

The apparatus of claim 15, wherein the delay unit is disposed along a path between a signal change unit having the high frequency inserter and the first scene change detector, and the signal detector, and further comprising:
a feedback line to provide the calculated time difference from the sync calculator back to the delay unit.

### Claim 23

An apparatus to maintain synchronization of video data, comprising:
an audio video synchronization unit, comprising:
a signal change unit to insert a synchronization mark into an audio signal of the video data when a scene changes in a video signal of the video data;
a post processing unit to perform one or more signal processing operations on the video signal and the audio signal that introduce a relative delay between the video signal and the audio signal, respectively; and
a resynchronization unit to resynchronize the video signal and the audio signal to account for the relative delay by calculating a time difference between when the scene changes and when the synchronization mark is detected in the audio signal and delaying an output audio signal and an output video signal based on the time difference.

### Claim 24 (depends on 23)

The apparatus of claim 23, further comprising:
an audio decoding apparatus to receive the video data including the video signal and the audio signal from an external source, to decode the video signal and the audio signal, and to provide the decoded video signal and the decoded audio signal to the audio video synchronization unit.

### Claim 25 (depends on 23)

The apparatus of claim 23, further comprising:
a display unit to output the video signal; and
a light sensor to detect the scene change in the video signal and to provide a time when the scene changes in the video signal to the audio video synchronization unit.

### Claim 26 (depends on 23)

The apparatus of claim 23, wherein the resynchronization unit comprises:
a signal detector to detect a time when the scene changes in the video signal and a time to when the synchronization mark is present in the audio signal;
a sync calculator to determine the time difference between when the scene changes in the video signal and when the synchronization mark is present in the audio signal; and
an audio delay unit to independently buffer the video signal and the audio signal to account for the determined time difference.

### Claim 27

An apparatus to resynchronize video data, comprising:
a signal receiving unit to receive unsynchronized video data including a video signal and an audio signal having a relative delay therebetween, and the audio signal having one or more synchronization marks inserted therein to indicate when a scene changes in the video signal; and
a synchronizing unit to synchronize the audio signal and the video signal to account for the relative delay by calculating a time difference between when the scene changes in the video signal and when the synchronization mark is detected in the audio signal and to delay an output audio signal and an output video signal according to the time difference.

### Claim 28 (depends on 27)

The apparatus of claim 27, further comprising:
a display unit to display the video signal;
a light sensor to detect a scene change by comparing luminance levels between consecutive frames of the displayed video signal; and
an audio detection unit to detect the one or more synchronization marks in the audio signal.

### Claim 29

An apparatus to insert synchronization information into video data, comprising:
a scene detecting unit to detect when a scene changes in a video signal of the video data; and
an inserting unit to insert a synchronization mark into an audio signal of the video data when the scene changes in the video signal.

### Claim 30 (depends on 29)

The apparatus of claim 29, wherein the scene detecting unit detects when the scene changes by determining a luminance level for each pixel in a current frame, determining a difference between the luminance level of each pixel in the current frame and a luminance level of each pixel in a previous frame, accumulating each luminance level difference between the current frame and the previous frame, and determining whether the accumulated difference exceeds a predetermined threshold.

### Claim 31

An apparatus to synchronize video data including a video signal and an audio signal, comprising:
a signal modification unit to modify an audio signal by adding an identifier thereto whenever a scene change is detected in a corresponding video signal;
a delay unit to delay the modified audio signal and the corresponding video signal for synchronization;
a post processing unit to perform process operations on the modified audio signal and the corresponding video signal and to output the processed audio signal and corresponding video signal; and
a resynchronization unit to control the delay unit to resynchronize the modified audio signal and corresponding video signal based on a delay therebetween caused by the post processing unit.

### Claim 32

A computer readable medium having executable code to synchronize audio and video signals, the medium comprising:
a first executable code to output a video data including a video signal and corresponding audio signal including a high frequency component having a predetermined pattern inserted therein to indicate when a scene change occurs in the video signal;
a second executable code to detect a scene change in the output video data and to detect the high frequency component having the predetermined pattern in the audio signal;
a third executable code to calculate a time difference between a time when the scene change is detected in the output video data and a time when the high frequency component having the predetermined pattern is detected in the audio signal; and
a fourth executable code to control delay times of the audio signal and the video signal according to the calculated time difference.
