# US 12,192,599 B2 · Asynchronous content analysis for synchronizing audio and video streams > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,599 B2 - Document ID: US12192599B2 - Status: Granted - Kind: B2 - Application number: 18/333,132 - Filed: 2023-06-12 - Published: 2024-12-12 - Granted: 2025-01-07 - Earliest claimed: 2023-06-12 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Indervir Singh Banipal; Shikhar Kwatra; Vijay Ekambaram; Hemant Kumar Sivaswamy - Web page: https://patentgrep.com/patent/US12192599B2 ## CPC - H04N 21/8547: involving timestamps for synchronizing content - H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/242: Synchronisation processes, e.g. processing of PCR [Programme Clock References] {(arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18)} - H04N 21/43072: {of multiple content streams on the same device} - H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Embodiments of the present disclosure provide systems and methods for synchronizing an unaligned audio stream and a corresponding unaligned video stream of real time streaming media. A non-limiting disclosed method comprises performing, using a video classifier, video reference point classification of a video stream based on an audio-video dataset; performing, using an audio classifier, audio reference point classification of the audio stream based on the audio-video dataset. The system correlates object related reference points in video segments of the video stream and in audio segments of the audio stream to identify a set of audio-video synchronization candidates. The system compares context of the set of audio-video synchronization candidates to identify an audio-video synchronization candidate to synchronize the audio stream and video stream based on reference point alignment. ## Claims ### Claim 1 A method comprising: performing, using a video classifier, video reference point classification of a video stream based on an audio-video dataset; wherein the audio-video dataset comprises audio-video data of objects causing a Doppler-effect sound in a corresponding audio stream; performing, using an audio classifier, audio reference point classification of the corresponding audio stream based on the audio-video dataset; identifying, based on the video reference point classification, a set of video segments comprising object related reference points in the video stream; wherein the object related reference points in the video stream include the Doppler-effect sound in the audio stream; identifying, based on the audio reference point classification, a set of audio segments comprising object related reference points of the Doppler-effect sound in the audio stream; correlating object related reference points in the video segments of the video stream and in the audio segments of the audio stream to identify a set of audio-video synchronization candidates; and comparing context of the set of the audio-video synchronization candidates to identify an audio-video synchronization candidate to synchronize the audio stream and the video stream. ### Claim 2 (depends on 1) The method of claim 1, wherein comparing context of the set of the audio-video synchronization candidates further comprises performing time slicing sliding window analysis to synchronize the audio stream and video stream. ### Claim 3 (depends on 2) The method of claim 2, wherein performing sliding window analysis to synchronize the audio stream and video stream further comprises separating a video component from an audio component and synchronizing with respect to the audio component, delaying the audio component with respect to the video component to make an alignment correction. ### Claim 4 (depends on 1) The method of claim 1, wherein comparing context of the set of the audio-video synchronization candidates further comprises analyzing Doppler frequency information within audio stream and video stream signals of the audio-video synchronization candidate to identify corresponding audio and video frequencies to synchronize the audio stream and video stream. ### Claim 5 (depends on 1) The method of claim 1, wherein performing, using the video classifier, video reference point classification of a video stream based on the audio-video dataset further comprises matching a set of video object related reference points with a set of audio object related reference points based on the audio-video dataset. ### Claim 6 (depends on 1) The method of claim 1, wherein comparing context of the set of the audio-video synchronization candidates further comprises analyzing content of the object related reference points in the video segments of the video stream and the object related reference points in the audio segments of the audio stream to identify context of the set of the audio-video synchronization candidates. ### Claim 7 (depends on 1) The method of claim 1, wherein comparing context of the set of the audio-video synchronization candidates to identify the audio-video synchronization candidate to synchronize the audio stream and video stream further comprises analyzing content of the object related reference points in the video segments of the video stream and the object related reference points in the audio segments of the audio stream to identify audio features of the set of the audio-video synchronization candidates. ### Claim 8 (depends on 7) The method of claim 7, further comprises matching the audio features of the set of the audio-video synchronization candidates to identify and compare the context of the set of the audio-video synchronization candidates to identify the audio-video synchronization candidate. ### Claim 9 (depends on 1) The method of claim 1, wherein performing, using an audio classifier, audio reference point classification of the corresponding audio stream based on the audio-video dataset further comprises identifying a set of audio reference points in the audio stream matching the Doppler-effect sound of the object related reference points in the video stream. ### Claim 10 (depends on 9) The method of claim 9, wherein identifying a set of audio reference points in the audio stream matching the Doppler-effect sound of the object related reference points in the video stream further comprises identifying context C1 from analyzing the audio stream and context C2 from analyzing the video stream, reducing the set of audio-video synchronization candidates to identify the audio-video synchronization candidate where context C1 equals context C2. ### Claim 11 A system, comprising: a processor; and a memory, wherein the memory includes a computer program product configured to perform operations for synchronizing an unaligned audio stream and a corresponding unaligned video stream of real time streaming media, the operations comprising: performing, using a video classifier, video reference point classification of a video stream based on an audio-video dataset; wherein the audio-video dataset comprises audio-video data of objects causing a Doppler-effect sound in a corresponding audio stream; performing, using an audio classifier, audio reference point classification of the corresponding audio stream based on the audio-video dataset; identifying, based on the video reference point classification, a set of video segments comprising object related reference points in the video stream; wherein object related reference points in the video stream include the Doppler-effect sound in the audio stream; identifying, based on the audio reference point classification, a set of audio segments comprising object related reference points of the Doppler-effect sound in the audio stream; correlating object related reference points in the video segments of the video stream and in the audio segments of the audio stream to identify a set of audio-video synchronization candidates; and comparing context of the set of the audio-video synchronization candidates to identify an audio-video synchronization candidate to synchronize the audio stream and the video stream. ### Claim 12 (depends on 11) The system of claim 11, wherein comparing context of the set of the audio-video synchronization candidates further comprises matching audio features of the set of the audio-video synchronization candidates to identify context of the set of the audio-video synchronization candidates. ### Claim 13 (depends on 11) The system of claim 11, wherein comparing context of the set of the audio-video synchronization candidates further comprises analyzing content of the object related reference points in the video segments of the video stream and in the audio segments of the audio stream to identify context of the set of the audio-video synchronization candidates. ### Claim 14 (depends on 11) The system of claim 11, wherein comparing context of the set of the audio-video synchronization candidates to identify the audio-video synchronization candidate to synchronize the audio stream and video stream further comprises analyzing content of the object related reference points in the video segments of the video stream and in the audio segments of the audio stream to identify audio features of the set of the audio-video synchronization candidates; and matching the audio features. ### Claim 15 (depends on 11) The system of claim 11, wherein identifying a set of object related reference points in the audio stream matching the Doppler-effect sound of the object related reference points in the video stream further comprises identifying context C1 from analyzing content of the audio stream and identifying context C2 from analyzing content of the video stream, identifying where context C1 equals context C2 for reducing the set of audio-video synchronization candidates to the audio-video synchronization candidate. ### Claim 16 A computer program product for synchronizing an unaligned audio stream and a corresponding unaligned video stream of real time streaming media, the computer program product comprising: a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation comprising: performing, using a video classifier, video reference point classification of a video stream based on an audio-video dataset; wherein the audio-video dataset comprises audio-video data of objects causing a Doppler-effect sound in a corresponding audio stream; performing, using an audio classifier, audio reference point classification of the corresponding audio stream based on the audio-video dataset; identifying, based on the video reference point classification, a set of video segments comprising object related reference points in the video stream; wherein object related reference points in the video stream include the Doppler-effect sound in the audio stream; identifying, based on the audio reference point classification, a set of audio segments comprising object related reference points of the Doppler-effect sound in the audio stream; correlating object related reference points in the video segments of the video stream and in the audio segments of the audio stream to identify a set of audio-video synchronization candidates; and comparing context of the set of the audio-video synchronization candidates to identify an audio-video synchronization candidate to synchronize the audio stream and the video stream. ### Claim 17 (depends on 16) The computer program product of claim 16, wherein comparing context of the set of the audio-video synchronization candidates further comprises matching audio features of the set of the audio-video synchronization candidates to identify context of the set of the audio-video synchronization candidates. ### Claim 18 (depends on 16) The computer program product of claim 16, wherein comparing context of the set of the audio-video synchronization candidates further comprises analyzing content of the object related reference points in the video segments of the video stream and in the audio segments of the audio stream to identify context of the set of the audio-video synchronization candidates. ### Claim 19 (depends on 16) The computer program product of claim 16, wherein comparing context of the set of the audio-video synchronization candidates to identify the audio-video synchronization candidate to synchronize the audio stream and video stream further comprises analyzing content of the object related reference points in the video segments of the video stream and in the audio segments of the audio stream to identify audio features of the set of the audio-video synchronization candidates; and matching the audio features. ### Claim 20 (depends on 16) The computer program product of claim 16, wherein identifying a set of object related reference points in the audio stream matching the Doppler-effect sound of the object related reference points in the video stream further comprises identifying context C1 from analyzing content of the audio stream and identifying context C2 from analyzing content of the video stream, identifying where context C1 equals context C2 for reducing the set of audio-video synchronization candidates to the audio-video synchronization candidate.