# US 2024/0205477 A1 · System and method for synchronizing live video with external data sources > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2024/0205477 A1 - Document ID: US20240205477A1 - Status: Published application - Kind: A1 - Application number: 18/067,804 - Filed: 2022-12-19 - Published: 2024-06-20 - Earliest claimed: 2022-12-19 - Assignees: Disney Enterprises, Inc. - Applicants: Disney Enterprises, Inc. - Inventors: Andrew Hyde; Geoffrey Booth; Sujal Shah; Jonathan Flanders; Ognjen Boras; Aaron LaBerge - Web page: https://patentgrep.com/patent/US20240205477A1 ## CPC - H04N 21/242: Synchronisation processes, e.g. processing of PCR [Programme Clock References] {(arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18)} - H04L 65/75: Media network packet handling - H04N 21/26258: {for generating a list of items to be played back in a given order, e.g. playlist, or scheduling item distribution according to such list (retrieval of multimedia data based on playlists G06F16/40)} - H04N 21/4305: {Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets (arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083)} - H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments} - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Methods and systems for synchronizing video with external data sources by a user device where the video is grouped into segment video files having video frames, includes establishing a persistent connection between the user device and a synchronization server, receiving a sync message associated with the beginning of a video segment to be played by the user device, the sync message having an initial recording wall clock time stamp, calculating by the user device a recording wall clock time for each frame in the segment using the initial wall clock time and a frame rate, downloading the video segment file for playing on the user device, receiving by the user device, external data having a wall clock time stamp, and displaying on the user device the external data having the same time stamp as the recording time stamp of the video frame being displayed. ## Claims ### Claim 1 A method for synchronizing streaming video with external data sources received by a user device, the video being grouped into segment video files having video frames, comprising: establishing a persistent connection between the user device and a synchronization server; receiving by the user device from the synchronization server via the persistent connection, a sync message associated with the beginning of a video segment having video frames to be played by the user device, the sync message having an initial recording wall clock time stamp corresponding to a first frame of the video segment; and calculating, by the user device, a recording wall clock time for each frame in the segment using the initial recording wall clock time stamp and a frame rate for the video segment. ### Claim 2 (depends on 1) The method of claim 1, further comprising: downloading to the user device, via a segment delivery connection between the user device and a segment server, separate from the persistent connection, the video segment file from the segment server, for playing on the user device; receiving by the user device external data having an event wall clock time, the external data for a given event wall clock time stamp being received by the user device before the video segment having a corresponding time stamp; and displaying on the user device, the external data having the same wall clock time stamp as the wall clock time stamp of the video frame being displayed, thereby providing realtime synchronized video and external data displayed on the user device. ### Claim 3 (depends on 1) The method of claim 1, wherein the frame rate is received from the sync message. ### Claim 4 (depends on 1) The method of claim 1, wherein the sync message further comprises at least one of: the frame rate for the video segment, a segment ID indicative of the video segment, and an airing ID indicative of the event associated with the streaming video. ### Claim 5 (depends on 1) The method of claim 1, further comprising storing by the user device the external data and at least one of: an associated event wall clock time, an associated video segment ID, and an associated video frame index number. ### Claim 6 (depends on 5) The method of claim 5, further comprising retrieving by the user device the stored external data when the event wall clock time associated with the video segment and video frame to be played next catches-up with the stored external data. ### Claim 7 (depends on 5) The method of claim 5, wherein the stored external data comprises at least one of: external data source identifier, time stamp, segment ID, frame index, external data content received, and external data logic/actions. ### Claim 8 (depends on 1) The method of claim 1, further comprising determining by the user device how to display the external data on the user device, based on external data logic. ### Claim 9 (depends on 1) The method of claim 1, wherein the establishing step comprises sending a sync message add request from the user device to the synchronization server to receive sync messages from the synchronization server. ### Claim 10 (depends on 1) The method of claim 1, wherein the sync message comprises information indicative of characteristics of the new segment file. ### Claim 11 (depends on 1) The method of claim 1, further comprising, after the displaying step, repeating all steps in the order shown, except for the establishing step, for each successive video segment file. ### Claim 12 (depends on 1) The method of claim 1, wherein the external data comprises at least one of: realtime event data, video data, and user-specific message data relating to the video being played. ### Claim 13 (depends on 1) The method of claim 1, wherein the persistent connection comprises websockets. ### Claim 14 (depends on 1) The method of claim 1, wherein the sync message is sent by the synchronization server simultaneously to all user devices that have established the persistent connection with the synchronization server. ### Claim 15 (depends on 1) The method of claim 1, further comprising receiving a manifest pointer from a manifest server indicative of the video segment file in the segment server. ### Claim 16 (depends on 15) The method of claim 15, further comprising requesting the manifest pointer by the user device from the manifest server. ### Claim 17 (depends on 15) The method of claim 15, further comprising updating a user device local manifest file with the manifest pointer from the manifest server, wherein at least a portion of the local manifest file is a copy of a source manifest file on the manifest server. ### Claim 18 (depends on 17) The method of claim 17, wherein the user device local manifest file comprises a top-level manifest file having pointers that point to at least one other manifest file having pointers indicative of the video segment file. ### Claim 19 (depends on 15) The method of claim 15, wherein the manifest pointer is indicative of at least one of: the location of the video segment file in the segment server and a second manifest file having information indicative of the video segment file. ### Claim 20 (depends on 1) The method of claim 1, wherein the video is live streaming video or pre-recorded video. ### Claim 21 (depends on 1) The method of claim 1, wherein the video segment file has a playing length of about 2 seconds, and the frame rate is about 60 frames/second. ### Claim 22 A method for synchronizing video with external data sources on a user device, the video grouped into segment video files having video frames, comprising: establishing a persistent connection between the user device and a synchronization server; receiving a sync message associated with the beginning of a video segment to be played by the user device, the sync message having at least an initial recording wall clock time stamp corresponding to a first frame of the video segment; determining by the user device, using the initial recording wall clock time stamp and a frame rate, a recording wall clock time for each video frame in the video segment; downloading the video segment file for playing on the user device; receiving by the user device, external data having an event wall clock time stamp; and synchronizing in realtime by the user device, the external data with the video segment file having the same time stamp. ### Claim 23 (depends on 22) The method of claim 22, wherein the sync message further comprises at least one of a segment ID indicative of the video segment and an airing ID indicative of the event associated with the streaming video. ### Claim 24 (depends on 22) The method of claim 22, further comprising storing by the user device the external data and at least one of: an associated event wall clock time, an associated video segment ID, and an associated video frame index number. ### Claim 25 (depends on 24) The method of claim 24, further comprising retrieving by the user device the stored external data when the event wall clock time associated with the video segment and video frame to be played next catches-up with the stored external data. ### Claim 26 (depends on 25) The method of claim 25, wherein the stored external data comprises at least one of: external data source identifier, time stamp, segment ID, frame index, external data content received, and external data logic/actions. ### Claim 27 (depends on 22) The method of claim 22, further comprising determining by the user device how to display the external data on the user device, based on external data logic. ### Claim 28 (depends on 22) The method of claim 22, wherein the establishing step comprises sending a sync message add request from the user device to the synchronization server to receive sync messages from the synchronization server. ### Claim 29 (depends on 22) The method of claim 22, wherein the sync message comprises information indicative of characteristics of the new segment file. ### Claim 30 (depends on 22) The method of claim 22, further comprising, after the synchronizing step, repeating all steps in the order shown, except for the establishing step, for each successive video segment file. ### Claim 31 (depends on 22) The method of claim 22, wherein the persistent connection comprises websockets. ### Claim 32 (depends on 22) The method of claim 22, wherein the sync message is sent by the synchronization server simultaneously to all user devices that have established the persistent connection with the synchronization server. ### Claim 33 (depends on 22) The method of claim 22, further comprising receiving a manifest pointer from a manifest server indicative of the video segment file in the segment server. ### Claim 34 (depends on 22) The method of claim 22, wherein the video is live streaming video or pre-recorded video. ### Claim 35 A method for synchronizing video with external data sources, comprising: receiving by a user device streaming encoded video content grouped into segment video files, each segment video file having a plurality of video frames and having an initial recording wall clock time stamp corresponding to a first frame of the video segment; downloading to the user device, via a segment delivery connection between the user device and a segment server, a video segment file from the segment server, for playing on the user device; extracting from the video segment file the initial recording wall clock time stamp of the video segment; calculating, by the user device, a recording wall clock time for each frame in the video segment using the initial recording wall clock time stamp and a frame rate for the video segment; receiving by the user device external data having an event wall clock time, the external data for a given event wall clock time stamp being received by the user device before the video segment having a corresponding time stamp; and displaying on the user device, the external data having the same wall clock time stamp as the wall clock time stamp of the video frame being displayed, thereby providing realtime synchronized video and external data displayed on the user device.