# US 2019/0356897 A1 · Correlation of video stream frame timestamps based on a system clock > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0356897 A1 - Document ID: US20190356897A1 - Status: Published application - Kind: A1 - Application number: 15/990,549 - Filed: 2018-05-25 - Published: 2019-11-21 - Earliest claimed: 2018-05-21 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Sathyanarayanan KARIVARADASWAMY - Web page: https://patentgrep.com/patent/US20190356897A1 ## CPC - H04N 13/167: Synchronising or controlling image signals - H04N 21/23608: {Remultiplexing multiplex streams, e.g. involving modifying time stamps or remapping the packet identifiers} - H04N 5/08: Separation of synchronising signals from picture signals - H04N 13/139: Format conversion, e.g. of frame-rate or size - H04N 21/242: Synchronisation processes, e.g. processing of PCR [Programme Clock References] {(arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18)} - H04N 21/4305: {Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets (arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083)} - H04L 43/106: using time related information in packets, e.g. by adding timestamps - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract The methods described herein are configured to correlate frames of a video stream to a system clock. A correlator application receives a frame of a video stream from a capture device via a bus interface, the frame including start frame and end frame timestamps based on a device clock of the capture device. A second start frame timestamp and second end frame timestamp are predicted based on previously received frames and a system clock. A skew value of the frame is calculated based on differences between the start frame timestamps and the end frame timestamps. Upon the calculated skew value exceeding a skew threshold, the frame is corrected to correlate to the predicted start frame and end frame timestamps. The corrected frame is then provided for display. Correcting timestamps of video stream frames in the described manner reduces jitter and enables accurate synchronization of multiple video streams. ## Claims ### Claim 1 A computerized display system for correlating timestamps of video frames from a capture device comprising: at least one processor; at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the at least one processor to: receive, by a correlator application, a frame of a video stream from a capture device via a bus interface, the frame including a first start frame timestamp and a first end frame timestamp based on a device clock associated with the capture device; predict, by the correlator application, based on a plurality of previously received frames of the video stream, a second start frame timestamp and a second end frame timestamp associated with the received frame, the second start frame timestamp and second end frame timestamp being based on a system clock associated with the display system; calculate, by the correlator application, a skew value of the received frame based on a difference between the first start frame timestamp and the second start frame timestamp, and a difference between the first end frame timestamp and the second end frame timestamp; upon the calculated skew value exceeding a skew threshold, correct, by the correlator application, the received frame to correlate to the second start frame timestamp and the second end frame timestamp; and provide the corrected frame for display. ### Claim 2 (depends on 1) The computerized system of claim 1, wherein predicting the second start frame timestamp includes maintaining a mapping of start frame timestamps of the plurality of previously received frames of the video stream. ### Claim 3 (depends on 1) The computerized system of claim 1, wherein predicting the second start frame timestamp and the second end frame timestamp includes determining an average frame length of the plurality of previously received frames of the video stream, wherein the predicted second start frame timestamp and second end frame timestamp are separated by the determined average frame length. ### Claim 4 (depends on 1) The computerized system of claim 1, wherein correcting the received frame includes clipping the received frame to fit the predicted second start frame timestamp and second end frame timestamp and re-timestamping the received frame with the predicted second start frame timestamp and second end frame timestamp. ### Claim 5 (depends on 1) The computerized system of claim 1, wherein the at least one memory comprises computer program code, the at least one memory and the computer program code configured to, with the at least one processor, further cause the at least one processor to combine the received frame with the plurality of received frames of the video stream. ### Claim 6 (depends on 1) The computerized system of claim 1, wherein the video stream is synchronized with another video stream; and the corrected frame is synchronized with a matching frame of the other video stream based on at least one of the second start frame timestamp and the second end frame timestamp. ### Claim 7 (depends on 6) The computerized system of claim 6, wherein the video stream is a visible light-based video stream and the other video stream is an infrared-based video stream. ### Claim 8 (depends on 6) The computerized system of claim 6, wherein the video stream is a visible-light based video stream capturing a scene from a first angle and the other video stream is a visible-light based stream capturing the scene from a second angle. ### Claim 9 (depends on 1) The computerized system of claim 1, wherein the video stream includes live video. ### Claim 10 (depends on 1) The computerized system of claim 1, wherein the bus interface is a universal serial bus (USB) interface. ### Claim 11 A computerized method for correlating timestamps of video frames from a capture device, the computerized method comprising: receiving, by a correlator application, a frame of a video stream from a capture device via a bus interface, the frame including a first start frame timestamp and a first end frame timestamp based on a device clock associated with the capture device; predicting, by the correlator application, based on a plurality of previously received frames of the video stream, a second start frame timestamp and a second end frame timestamp associated with the received frame, the second start frame timestamp and second end frame timestamp being based on a system clock associated with a computing device of the correlator application; calculating, by the correlator application, a skew value of the received frame based on a difference between the first start frame timestamp and the second start frame timestamp, and a difference between the first end frame timestamp and the second end frame timestamp; upon the calculated skew value exceeding a skew threshold, correcting, by the correlator application, the received frame to correlate to the second start frame timestamp and the second end frame timestamp; combining the received frame with the plurality of received frames of the video stream; and providing the video stream for display. ### Claim 12 (depends on 11) The computerized method of claim 11, wherein predicting the second start frame timestamp includes maintaining a mapping of start frame timestamps of the plurality of previously received frames of the video stream. ### Claim 13 (depends on 11) The computerized method of claim 11, wherein predicting the second start frame timestamp and the second end frame timestamp includes determining an average frame length of the plurality of previously received frames of the video stream, wherein the predicted second start frame timestamp and second end frame timestamp are separated by the determined average frame length. ### Claim 14 (depends on 11) The computerized method of claim 11, wherein correcting the received frame includes clipping the received frame to fit the predicted second start frame timestamp and second end frame timestamp and re-timestamping the received frame with the predicted second start frame timestamp and second end frame timestamp. ### Claim 15 (depends on 11) The computerized method of claim 11, wherein the video stream is synchronized with another video stream; and the corrected frame is synchronized with a matching frame of the other video stream based on at least one of the second start frame timestamp and the second end frame timestamp. ### Claim 16 (depends on 15) The computerized method of claim 15, wherein the video stream is a visible light-based video stream and the other video stream is a depth-based video stream. ### Claim 17 (depends on 15) The computerized method of claim 15, wherein the video stream is a visible-light based video stream capturing a scene from a first angle and the other video stream is a visible-light based stream capturing the scene from a second angle. ### Claim 18 One or more computer storage media having computer-executable instructions for correlating timestamps of video frames from capture devices that, upon execution by a processor, cause the processor to at least: receive a frame of a video stream from a capture device via a bus interface, the frame including a first start frame timestamp and a first end frame timestamp based on a device clock associated with the capture device; predict based on a plurality of previously received frames of the video stream, a second start frame timestamp and a second end frame timestamp associated with the received frame, the second start frame timestamp and second end frame timestamp being based on a system clock associated with the processor; calculate a skew value of the received frame based on a difference between the first start frame timestamp and the second start frame timestamp, and a difference between the first end frame timestamp and the second end frame timestamp; upon the calculated skew value exceeding a skew threshold, correct the received frame to correlate to the second start frame timestamp and the second end frame timestamp; and provide the corrected frame for display. ### Claim 19 (depends on 18) The one or more computer storage media of claim 18, wherein the computer-executable instructions, upon execution by a processor, further cause the processor to at least: receive frames of a second video stream from a second capture device; timestamp the received frames of the second video stream; combine the video stream with the second video stream; and wherein the corrected frame is synchronized with a matching frame of the second video stream. ### Claim 20 (depends on 19) The one or more computer storage media of claim 19, wherein the video stream is a visible light-based video stream and the second video stream is a depth camera-based video stream; and wherein combining the video stream with the second video stream creates a three-dimensional correlated video stream.