Real-time latency measurement of video streams

A system for identifying real-time latency of video streams, comprising:

• a computing device programmed to
• buffer frames of a video stream into a first buffer;
• buffer frames of a video stream into a second buffer;
• identify a control group as a subset of frames of the second buffer;
• compute correlations of extracted features of the control group to extracted features of successive windows of frames of the first buffer, the extracted features being based on spatial information and temporal information of the video stream buffered into the first buffer and the video stream buffered into the second buffer, including to
• perform calculations on the video stream buffered into the first buffer to identify valid frames for which the spatial information exceeds a spatial information threshold level of information and the temporal information exceeds a temporal information threshold level of information;
• construct, as the control group, a control group pattern vector indicating a pattern of valid frames;
• using a sliding window approach, advance a window through frames of the video stream buffered into the second buffer to extract temporal and spatial features and to compute differences and correlation between the pattern of valid frames of the video stream buffered into the first buffer and corresponding frames of the window in the video stream buffered into the second buffer according to the temporal and spatial features, such that a sum of difference values Art each window is computed as a group difference for the respective window;
• determine a frame offset between the video stream buffered into the first buffer and the video stream buffered into the second buffer according to the group difference and the correlation;
• and
• compute a latency between the video stream buffered into the first buffer and the video stream buffered into the second buffer according to reading times of the video stream buffered into the first buffer and the video stream buffered into the second buffer, frame rates of the video stream buffered into the first buffer and the video stream buffered into the second buffer, and the frame offset between the video stream buffered into the first buffer and the video stream buffered into the second buffer.

A non-transitory computer-readable medium comprising instructions for identifying real-time latency of video streams, that when executed by a processor of a computing device, cause the computing device to:

• buffer frames of a video stream into a first buffer;
• buffer frames of a video stream into a second buffer;
• identify a control group as a subset of frames of the second buffer;
• compute correlations of extracted features of the control group to extracted features of successive windows of frames of the first buffer, the extracted features being based on spatial information and temporal information of the video stream buffered into the first buffer and the video stream buffered into the second buffer, including to
• perform calculations on the video stream buffered into the first buffer to identify valid frames for which the spatial information exceeds a spatial information threshold level of information and the temporal information exceeds a temporal information threshold level of information;
• construct, as the control group, a control group pattern vector indicating a pattern of valid frames;
• using a sliding window approach, advance a window through frames of the video stream buffered into the second buffer to extract temporal and spatial features and to compute differences and correlation between the pattern of valid frames of the video stream buffered into the first buffer and corresponding frames of the window in the video stream buffered into the second buffer according to the temporal and spatial features, such that a sum of difference values in each window is computed as a group difference for the respective window;
• determine a frame offset between the video stream buffered into the first buffer and the video stream buffered into the second buffer according to the group difference and the correlation;
• and
• compute a latency between the video stream buffered into the first buffer and the video stream buffered into the second buffer according to reading times of the video stream buffered into the first buffer and the video stream buffered into the second buffer, frame rates of the video stream buffered into the first buffer and the video stream buffered into the second buffer, and the frame offset between the video stream buffered into the first buffer and the video stream buffered into the second buffer.