# US 11,451,830 B2 · Reducing latency in video encoding and decoding > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,830 B2 - Document ID: US11451830B2 - Status: Granted - Kind: B2 - Application number: 17/143,782 - Filed: 2021-01-07 - Published: 2021-04-29 - Granted: 2022-09-20 - Earliest claimed: 2011-06-30 - Assignees: Microsoft Technology Licensing, LLC - Applicants: Microsoft Technology Licensing, LLC - Inventors: Gary J. Sullivan - Web page: https://patentgrep.com/patent/US11451830B2 ## CPC - H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards - H04N 19/103: Selection of coding mode or of prediction mode - H04N 19/15: by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer - H04N 19/172: the region being a picture, frame or field - H04N 19/31: in the temporal domain - H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/88: involving rearrangement of data among different coding units, e.g. shuffling, interleaving, scrambling or permutation of pixel data or permutation of transform coefficient data among different blocks - H04N 19/184: the unit being bits, e.g. of the compressed video stream - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Techniques and tools for reducing latency in video encoding and decoding by constraining latency due to reordering of video frames, and by indicating the constraint on frame reordering latency with one or more syntax elements that accompany encoded data for the video frames. For example, a real-time communication tool with a video encoder sets a syntax element that indicates a constraint on frame reordering latency, which is consistent with inter-frame dependencies between multiple frames of a video sequence, then outputs the syntax element. A corresponding real-time communication tool with a video decoder receives the syntax element that indicates the constraint on frame reordering latency, determines the constraint on frame reordering latency based on the syntax element, and uses the constraint on frame reordering latency to determine when a reconstructed frame is ready for output (in terms of output order). ## Claims ### Claim 1 In a computing system that implements a video decoder, a method comprising: receiving encoded data for at least part of a video sequence, the encoded data including one or more syntax elements that indicate a constraint on frame reordering latency; decoding at least part of the encoded data to reconstruct a frame of the video sequence; storing the reconstructed frame in a decoded picture buffer (“DPB”); determining when the reconstructed frame in the DPB is ready for output, the reconstructed frame in the DPB being ready for output if latency status of the reconstructed frame in the DPB is equal to the constraint on frame reordering latency upon: determining the latency status of the reconstructed frame in the DPB; and comparing the latency status of the reconstructed frame in the DPB to the constraint on frame reordering latency; and outputting the reconstructed frame. ### Claim 2 (depends on 1) The method of claim 1, wherein the latency status of the reconstructed frame in the DPB indicates a difference between output order and coded order for the reconstructed frame in the DPB. ### Claim 3 (depends on 1) The method of claim 1, wherein the reconstructed frame is output before the DPB is full, the constraint on frame reordering latency being lower than size of the DPB. ### Claim 4 (depends on 1) The method of claim 1, wherein frames of the video sequence are organized according to a temporal hierarchy, and wherein different constraints on frame reordering latencies apply for different temporal layers of the temporal hierarchy, the method further comprising selecting, as the constraint on frame reordering latency, one of the different constraints on frame reordering latencies depending on temporal resolution of the output. ### Claim 5 (depends on 1) The method of claim 1, further comprising: negotiating the constraint on frame reordering latency for an interactive real-time communication scenario. ### Claim 6 (depends on 1) The method of claim 1, wherein, according to the one or more syntax elements, the constraint on frame reordering latency is based on a value indicating a size of the DPB and a value indicating a count of frames relative to the size of the DPB. ### Claim 7 A video decoding system comprising: memory configured to receive encoded data for at least part of a video sequence, the encoded data including one or more syntax elements that indicate a constraint on frame reordering latency; a decoded picture buffer (“DPB”); and a decoder configured to: decode at least part of the encoded data to reconstruct a frame of the video sequence; store, in the DPB, the reconstructed frame; determine when the reconstructed frame in the DPB is ready for output, the reconstructed frame in the DPB being ready for output if latency status of the reconstructed frame in the DPB is equal to the constraint on frame reordering latency upon: determining the latency status of the reconstructed frame in the DPB; and comparing the latency status of the reconstructed frame in the DPB to the constraint on frame reordering latency; and output the reconstructed frame. ### Claim 8 (depends on 7) The video decoding system of claim 7, wherein the latency status of the reconstructed frame in the DPB indicates a difference between output order and coded order for the reconstructed frame in the DPB. ### Claim 9 (depends on 7) The video decoding system of claim 7, wherein the DPB has a size, and wherein the constraint on frame reordering latency is lower than the size of the DPB. ### Claim 10 (depends on 7) The video decoding system of claim 7, wherein frames of the video sequence are organized according to a temporal hierarchy, and wherein different constraints on frame reordering latencies apply for different temporal layers of the temporal hierarchy, the decoder being further configured to select, as the constraint on frame reordering latency, one of the different constraints on frame reordering latencies depending on temporal resolution of the output. ### Claim 11 (depends on 7) The video decoding system of claim 7, wherein, according to the one or more syntax elements, the constraint on frame reordering latency is based on a value indicating a size of the DPB and a value indicating a count of frames relative to the size of the DPB. ### Claim 12 Computer-readable tangible storage having stored thereon encoded data organized in a bitstream for at least part of a video sequence, the encoded data including one or more syntax elements that indicate a constraint on frame reordering latency, and the encoded data being organized to facilitate playback, in a computing system that implements a video decoder, by operations comprising: decoding at least part of the encoded data to reconstruct a frame of the video sequence; storing the reconstructed frame in a decoded picture buffer (“DPB”); determining when the reconstructed frame in the DPB is ready for output, the reconstructed frame in the DPB being ready for output if latency status of the reconstructed frame in the DPB is equal to the constraint on frame reordering latency upon: determining the latency status of the reconstructed frame in the DPB; and comparing the latency status of the reconstructed frame in the DPB to the constraint on frame reordering latency; and outputting the reconstructed frame. ### Claim 13 (depends on 12) The computer-readable tangible storage of claim 12, wherein the latency status of the reconstructed frame in the DPB indicates a difference between output order and coded order for the reconstructed frame in the DPB. ### Claim 14 (depends on 12) The computer-readable tangible storage of claim 12, wherein the reconstructed frame is output before the DPB is full, the constraint on frame reordering latency being lower than size of the DPB. ### Claim 15 (depends on 12) The computer-readable tangible storage of claim 12, wherein frames of the video sequence are organized according to a temporal hierarchy, and wherein different constraints on frame reordering latencies apply for different temporal layers of the temporal hierarchy, the operations further comprising selecting, as the constraint on frame reordering latency, one of the different constraints on frame reordering latencies depending on temporal resolution of the output. ### Claim 16 (depends on 12) The computer-readable tangible storage of claim 12, the operations further comprising: negotiating the constraint on frame reordering latency for an interactive real-time communication scenario. ### Claim 17 (depends on 12) The computer-readable tangible storage of claim 12, wherein, according to the one or more syntax elements, the constraint on frame reordering latency is based on a value indicating a size of the DPB and a value indicating a count of frames relative to the size of the DPB. ### Claim 18 In a computing system that implements a video encoder, a method comprising: receiving frames of a video sequence; encoding the frames of the video sequence, thereby producing encoded data; and outputting the encoded data as part of a bitstream, the encoded data including one or more syntax elements that indicate a constraint on frame reordering latency, thereby enabling a video decoder to perform operations to decode at least part of the encoded data to reconstruct one of the frames of the video sequence, store the reconstructed frame in a decoded picture buffer (“DPB”), determine when the reconstructed frame in the DPB is ready for output, and output the reconstructed frame, the reconstructed frame in the DPB being ready for output if latency status of the reconstructed frame in the DPB is equal to the constraint on frame reordering latency upon determining the latency status of the reconstructed frame in the DPB and comparing the latency status of the reconstructed frame in the DPB to the constraint on frame reordering latency. ### Claim 19 (depends on 18) The method of claim 18, wherein the latency status of the reconstructed frame in the DPB indicates a difference between output order and coded order for the reconstructed frame in the DPB. ### Claim 20 (depends on 18) The method of claim 18, further comprising: negotiating the constraint on frame reordering latency for an interactive real-time communication scenario.