# US 2020/0404337 A1 · Reducing latency in video encoding and decoding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0404337 A1
- Document ID: US20200404337A1
- Status: Published application
- Kind: A1
- Application number: 17/009,918
- Filed: 2020-09-02
- Published: 2020-12-24
- Earliest claimed: 2011-06-30
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Gary J. Sullivan
- Web page: https://patentgrep.com/patent/US20200404337A1

## CPC

- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N 19/103: Selection of coding mode or of prediction mode
- H04N 19/15: by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
- H04N 19/172: the region being a picture, frame or field
- H04N 19/31: in the temporal domain
- H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/88: involving rearrangement of data among different coding units, e.g. shuffling, interleaving, scrambling or permutation of pixel data or permutation of transform coefficient data among different blocks
- H04N 19/184: the unit being bits, e.g. of the compressed video stream
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Techniques and tools for reducing latency in video encoding and decoding by constraining latency due to reordering of video frames, and by indicating the constraint on frame reordering latency with one or more syntax elements that accompany encoded data for the video frames. For example, a real-time communication tool with a video encoder sets a syntax element that indicates a constraint on frame reordering latency, which is consistent with inter-frame dependencies between multiple frames of a video sequence, then outputs the syntax element. A corresponding real-time communication tool with a video decoder receives the syntax element that indicates the constraint on frame reordering latency, determines the constraint on frame reordering latency based on the syntax element, and uses the constraint on frame reordering latency to determine when a reconstructed frame is ready for output (in terms of output order).

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

In a computing system that implements a video decoder, a method comprising:
receiving, at the video decoder, encoded data in a bitstream for at least some frames of a video sequence, the encoded data including:
one or more first syntax elements that indicate a maximum size of frame memory for reordering, wherein the maximum size of frame memory for reordering is expressed in terms of a maximum count of frames that can precede any frame of the video sequence in coded order but follow that frame in output order; and
one or more second syntax elements, different than the one or more first syntax elements, that indicate a constraint on frame reordering latency, wherein the constraint on frame reordering latency is expressed in terms of a maximum count of frames that can precede any frame of the video sequence in output order but follow that frame in coded order;
decoding, with the video decoder, the encoded data, including:
decoding the one or more first syntax elements;
decoding the one or more second syntax elements; and
decoding at least some of the encoded data to reconstruct one of the at least some frames of the video sequence; and
outputting the reconstructed frame.

### Claim 22 (depends on 21)

The method of claim 21, wherein, in the encoded data, the one or more first syntax elements and the one or more second syntax elements have been encoded using unsigned exponential-Golomb coding.

### Claim 23 (depends on 21)

The method of claim 21, wherein the at least some frames of the video sequence are organized according to a temporal hierarchy, wherein different ones of the one or more second syntax elements indicate different constraints on frame reordering latency for different temporal layers of the temporal hierarchy, the method further comprising selecting one of the different constraints on frame reordering latency depending on temporal resolution of output.

### Claim 24 (depends on 21)

The method of claim 21, wherein the encoded data further includes one or more third syntax elements, different than the one or more first syntax elements and different than the one or more second syntax elements, that indicate a maximum size of a decoded picture buffer, and wherein the decoding the encoded data further includes:
decoding the one or more third syntax elements.

### Claim 25 (depends on 21)

The method of claim 21, wherein the one or more first syntax elements and the one or more second syntax elements are signaled as part of a sequence parameter set or media metadata.

### Claim 26 (depends on 21)

The method of claim 21, wherein one possible value of the one or more second syntax elements indicates the constraint on frame reordering latency is undefined or has a default value, and wherein other possible values of the one or more second syntax elements indicate an integer count for the constraint on frame reordering latency.

### Claim 27 (depends on 21)

The method of claim 21, wherein a value of the one or more second syntax elements indicates an integer count for the constraint on frame reordering latency relative to the maximum size of frame memory for reordering.

### Claim 28 (depends on 27)

The method of claim 27, wherein the constraint on frame reordering latency can be determined as the maximum count for the maximum size of frame memory for reordering plus the integer count for the constraint on frame reordering latency minus 1.

### Claim 29

A computer system comprising a processor and memory, wherein the computer system implements a video encoder configured to perform operations comprising:
with the video encoder, encoding at least some frames of a video sequence, thereby producing encoded data, including:
encoding one or more first syntax elements that indicate a maximum size of frame memory for reordering, wherein the maximum size of frame memory for reordering is expressed in terms of a maximum count of frames that can precede any frame of the video sequence in coded order but follow that frame in output order; and
encoding one or more second syntax elements, different than the one or more first syntax elements, that indicate a constraint on frame reordering latency, wherein the constraint on frame reordering latency is expressed in terms of a maximum count of frames that can precede any frame of the video sequence in output order but follow that frame in coded order; and
encoding the at least some frames of the video sequence, wherein the encoding the at least some frames uses inter-frame dependencies that are consistent with the constraint on frame reordering latency; and
outputting, from the video encoder, the encoded data as part of a bitstream, the encoded data including the one or more first syntax elements and the one or more second syntax elements, thereby facilitating determination of when reconstructed frames are ready for output in terms of output order.

### Claim 30 (depends on 29)

The computer system of claim 29, wherein the encoding the one or more first syntax elements and the encoding the one or more second syntax elements use unsigned exponential-Golomb coding.

### Claim 31 (depends on 29)

The computer system of claim 29, wherein the at least some frames of the video sequence are organized according to a temporal hierarchy, wherein different ones of the one or more second syntax elements indicate different constraints on frame reordering latency for different temporal layers of the temporal hierarchy, the operations further comprising selecting one of the different constraints on frame reordering latency depending on temporal resolution of the output.

### Claim 32 (depends on 29)

The computer system of claim 29, wherein the encoding the at least some frames further includes:
encoding one or more third syntax elements, different than the one or more first syntax elements and different than the one or more second syntax elements, that indicate a maximum size of a decoded picture buffer, the encoded data further including the one or more third syntax elements.

### Claim 33 (depends on 29)

The computer system of claim 29, wherein the one or more first syntax elements and the one or more second syntax elements are signaled as part of a sequence parameter set or media metadata.

### Claim 34 (depends on 29)

The computer system of claim 29, wherein one possible value of the one or more second syntax elements indicates the constraint on frame reordering latency is undefined or has a default value, and wherein other possible values of the one or more second syntax elements indicate an integer count for the constraint on frame reordering latency.

### Claim 35 (depends on 29)

The computer system of claim 29, wherein a value of the one or more second syntax elements indicates an integer count for the constraint on frame reordering latency relative to the maximum size of frame memory for reordering.

### Claim 36 (depends on 35)

The computer system of claim 35, wherein the constraint on frame reordering latency can be determined as the maximum count for the maximum size of frame memory for reordering plus the integer count for the constraint on frame reordering latency minus 1.

### Claim 37

One or more computer-readable media having stored thereon encoded data in a bitstream for at least some frames of a video sequence, the encoded data including (a) one or more first syntax elements that indicate a maximum size of frame memory for reordering, wherein the maximum size of frame memory for reordering is expressed in terms of a maximum count of frames that can precede any frame of the video sequence in coded order but follow that frame in output order, and (b) one or more second syntax elements, different than the one or more first syntax elements, that indicate a constraint on frame reordering latency, wherein the constraint on frame reordering latency is expressed in terms of a maximum count of frames that can precede any frame of the video sequence in output order but follow that frame in coded order, and wherein the encoded data is organized to facilitate decoding the encoded data by operations comprising, with a video decoder:
decoding the one or more first syntax elements;
decoding the one or more second syntax elements;
decoding at least some of the encoded data to reconstruct one of the at least some frames of the video sequence; and
outputting the reconstructed frame.

### Claim 38 (depends on 37)

The one or more computer-readable media of claim 37, wherein, in the encoded data, the one or more first syntax elements and the one or more second syntax elements have been encoded using unsigned exponential-Golomb coding.

### Claim 39 (depends on 37)

The one or more computer-readable media of claim 37, wherein the encoded data further includes one or more third syntax elements, different than the one or more first syntax elements and different than the one or more second syntax elements, that indicate a maximum size of a decoded picture buffer, and wherein the operations further include decoding the one or more third syntax elements.

### Claim 40 (depends on 37)

The one or more computer-readable media of claim 37, wherein a value of the one or more second syntax elements indicates an integer count for the constraint on frame reordering latency relative to the maximum size of frame memory for reordering, and wherein the constraint on frame reordering latency can be determined as the maximum count for the maximum size of frame memory for reordering plus the integer count for the constraint on frame reordering latency minus 1.
