# US 11,122,321 B1 · Stream synchronization using an automated digital clapperboard

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,122,321 B1
- Document ID: US11122321B1
- Status: Granted
- Kind: B1
- Application number: 16/840,679
- Filed: 2020-04-06
- Published: 2021-10-07
- Granted: 2021-09-14
- Earliest claimed: 2020-04-06
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Keith Gregory Frost; Stephen Arthur Boxwell; Kyle Matthew Brake; Stanley John Vernier
- Web page: https://patentgrep.com/patent/US11122321B1

## CPC

- H04N 21/242: Synchronisation processes, e.g. processing of PCR [Programme Clock References] {(arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18)}
- H04N 21/4307: {Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen}
- H04N 21/2187: Live feed
- H04N 21/43072: {of multiple content streams on the same device}
- H04N 21/4333: {Processing operations in response to a pause request}
- H04N 21/8106: {involving special audio data, e.g. different tracks for different languages}
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- H04N 21/854: Content authoring
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An approach is provided in which the approach receives a media stream comprising an audio stream and a video stream. The audio stream includes a set of audio tones inserted at a set of predetermined frame intervals, and the video stream includes a set of predetermined colors inserted into a set of predetermined pixel locations at the predetermined frame intervals. The approach determines that an audio frame in the audio stream at which the set of audio tones are inserted is misaligned in time with a video frame in the video stream at which the set of predetermined colors are included in the set of predetermined pixel locations. The approach synchronizes the audio stream with the video stream in response to the determination and aligns in time the audio frame with the video frame at one of the predetermined frame intervals.

## Claims

### Claim 1

A method implemented by an information handling system that includes a memory and a processor, the method comprising:
receiving, at a receiving device, a media stream comprising a metadata header, an audio stream and a video stream;
extracting, from the metadata header, synchronization information comprising a set of audio tone values corresponding to a set of inaudible audio tones, a set of color values corresponding to a set of colors, and a set of pixel location values corresponding to a set of pixel locations, wherein the audio stream comprises the set of inaudible audio tones inserted at a set of predetermined frame intervals, and wherein the video stream comprises the set of colors inserted at the set of pixel locations at the predetermined frame intervals;
in response to extracting the synchronization information, determining that a selected audio frame in the audio stream at which the set of inaudible audio tones corresponding to the extracted set of audio tone values are inserted is misaligned in time with a selected video frame in the video stream at which the set of colors corresponding to the extracted set of color values are inserted at the set of pixel locations corresponding to the extracted set of pixel location values; and
synchronizing the audio stream with the video stream at one of the set of predetermined frame intervals in response to the determination.

### Claim 2 (depends on 1)

The method of claim 1 further comprising:
detecting, at a first point in time, that the selected audio frame includes at least one inaudible tone in the set of inaudible audio tones; and
pausing the audio stream in response to determining that a first video frame located at the first point in time fails to include a first one of the set colors at a corresponding first one of the set of pixel locations.

### Claim 3 (depends on 2)

The method of claim 2 further comprising:
detecting, at a second point in time, that a second video frame includes the first one of the set colors at the corresponding first one of the set of pixel locations; and
unpausing the audio stream at the second point in time.

### Claim 4 (depends on 1)

The method of claim 1 further comprising:
detecting, at a first point in time, that the selected video frame includes a first one of the set of colors at a corresponding first one of the set of pixel locations; and
pausing the video stream in response to determining that a first audio frame located at the first point in time fails to include at least one inaudible tone in the set of inaudible audio tones.

### Claim 5 (depends on 4)

The method of claim 4 further comprising:
detecting, at a second point in time, that a second audio frame includes the at least one inaudible tone; and
unpausing the video stream at the second point in time.

### Claim 6 (depends on 1)

The method of claim 1 further comprising:
creating the metadata header at a transmitting device;
injecting, at the transmitting device, the set of inaudible audio tones into the audio stream at the predetermined frame intervals;
injecting, at the transmitting device, the set of colors at the set of pixel locations into the video stream at the predetermined frame intervals; and
sending the metadata header with the injected audio stream and the injected video stream to the receiving device.

### Claim 7

An information handling system comprising:
one or more processors;
a memory coupled to at least one of the processors;
a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions of:
receiving, at a receiving device, a media stream comprising a metadata header, an audio stream and a video stream;
extracting, from the metadata header, synchronization information comprising a set of audio tone values corresponding to a set of inaudible audio tones, a set of color values corresponding to a set of colors, and a set of pixel location values corresponding to a set of pixel locations, wherein the audio stream comprises the set of inaudible audio tones inserted at a set of predetermined frame intervals, and wherein the video stream comprises the set of colors inserted at the set of pixel locations at the predetermined frame intervals;
in response to extracting the synchronization information, determining that a selected audio frame in the audio stream at which the set of inaudible audio tones corresponding to the extracted set of audio tone values are inserted is misaligned in time with a selected video frame in the video stream at which the set of colors corresponding to the extracted set of color values are inserted at the set of pixel locations corresponding to the extracted set of pixel location values; and
synchronizing the audio stream with the video stream at one of the set of predetermined frame intervals in response to the determination.

### Claim 8 (depends on 7)

The information handling system of claim 7 wherein the processors perform additional actions comprising:
detecting, at a first point in time, that the selected audio frame includes at least one inaudible tone in the set of inaudible audio tones; and
pausing the audio stream in response to determining that a first video frame located at the first point in time fails to include a first one of the set colors at a corresponding first one of the set of pixel locations.

### Claim 9 (depends on 8)

The information handling system of claim 8 wherein the processors perform additional actions comprising:
detecting, at a second point in time, that a second video frame includes the first one of the set colors at the corresponding first one of the set of pixel locations; and
unpausing the audio stream at the second point in time.

### Claim 10 (depends on 7)

The information handling system of claim 7 wherein the processors perform additional actions comprising:
detecting, at a first point in time, that the selected video frame includes a first one of the set of colors at a corresponding first one of the set of pixel locations; and
pausing the video stream in response to determining that a first audio frame located at the first point in time fails to include at least one inaudible tone in the set of inaudible audio tones.

### Claim 11 (depends on 10)

The information handling system of claim 10 wherein the processors perform additional actions comprising:
detecting, at a second point in time, that a second audio frame includes the at least one inaudible tone; and
unpausing the video stream at the second point in time.

### Claim 12 (depends on 7)

The information handling system of claim 7 wherein the processors perform additional actions comprising:
creating the metadata header at a transmitting device;
injecting, at the transmitting device, the set of inaudible audio tones into the audio stream at the predetermined frame intervals;
injecting, at the transmitting device, the set of colors at the set of pixel locations into the video stream at the predetermined frame intervals; and
sending the metadata header with the injected audio stream and the injected video stream to the receiving device.

### Claim 13

A computer program product stored in a computer readable storage medium, comprising computer program code that, when executed by an information handling system, causes the information handling system to perform actions comprising:
receiving, at a receiving device, a media stream comprising a metadata header, an audio stream and a video stream;
extracting, from the metadata header, synchronization information comprising a set of audio tone values corresponding to a set of inaudible audio tones, a set of color values corresponding to a set of colors, and a set of pixel location values corresponding to a set of pixel locations, wherein the audio stream comprises the set of inaudible audio tones inserted at a set of predetermined frame intervals, and wherein the video stream comprises the set of colors inserted at the set of pixel locations at the predetermined frame intervals;
in response to extracting the synchronization information, determining that a selected audio frame in the audio stream at which the set of inaudible audio tones corresponding to the extracted set of audio tone values are inserted is misaligned in time with a selected video frame in the video stream at which the set of colors corresponding to the extracted set of color values are inserted at the set of pixel locations corresponding to the extracted set of pixel location values; and
synchronizing the audio stream with the video stream at one of the set of predetermined frame intervals in response to the determination.

### Claim 14 (depends on 13)

The computer program product of claim 13 wherein the information handling system performs further actions comprising:
detecting, at a first point in time, that the selected audio frame includes at least one inaudible tone in the set of inaudible audio tones; and
pausing the audio stream in response to determining that a first video frame located at the first point in time fails to include a first one of the set colors at a corresponding first one of the set of pixel locations.

### Claim 15 (depends on 14)

The computer program product of claim 14 wherein the information handling system performs further actions comprising:
detecting, at a second point in time, that a second video frame includes the first one of the set colors at the corresponding first one of the set of pixel locations; and
unpausing the audio stream at the second point in time.

### Claim 16 (depends on 13)

The computer program product of claim 13 wherein the information handling system performs further actions comprising:
detecting, at a first point in time, that the selected video frame includes a first one of the set of colors at a corresponding first one of the set of pixel locations; and
pausing the video stream in response to determining that a first audio frame located at the first point in time fails to include at least one inaudible tone in the set of inaudible audio tones.

### Claim 17 (depends on 16)

The computer program product of claim 16 wherein the information handling system performs further actions comprising:
detecting, at a second point in time, that a second audio frame includes the at least one inaudible tone and
unpausing the video stream at the second point in time.
