# US 2010/0135412 A1 · Media coding for loss recovery with remotely predicted data units

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2010/0135412 A1
- Document ID: US20100135412A1
- Status: Published application
- Kind: A1
- Application number: 12/699,354
- Filed: 2010-02-03
- Published: 2010-06-03
- Earliest claimed: 1999-03-12
- Assignees: Microsoft Corporation
- Inventors: Albert Szu-chi Wang; Ming-Chieh Lee
- Web page: https://patentgrep.com/patent/US20100135412A1

## CPC

- H04N 19/107: between spatial and temporal predictive coding, e.g. picture refresh
- G10L 19/005: Correction of errors induced by the transmission channel, if related to the coding algorithm
- H04L 65/70: Media network packetisation
- H04N 19/114: Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames (H04N19/107 takes precedence)
- H04N 19/124: Quantisation
- H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference
- H04N 19/166: concerning the amount of transmission errors, e.g. bit error rate [BER]
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/177: the unit being a group of pictures [GOP]
- H04N 19/20: using video object coding
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/50: using predictive coding (H04N19/61 takes precedence)
- H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109)
- H04N 19/513: Processing of motion vectors
- H04N 19/573: Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
- H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
- H04N 19/58: Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one (H04N19/23 takes precedence)
- H04N 19/593: involving spatial prediction techniques
- H04N 19/61: in combination with predictive coding
- H04N 19/89: involving methods or arrangements for detection of transmission errors at the decoder
- H04N 19/91: Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
- H04N 21/6377: directed to server {(one-way streaming services wherein the source is controlled by the destination H04L65/613)}
- H04N 21/658: Transmission by the client directed to the server
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

An improved loss recovery method for coding streaming media classifies each data unit in the media stream as an independent data unit (I unit), a remotely predicted unit (R unit) or a predicted data unit (P unit). Each of these units is organized into independent segments having an I unit, multiple P units and R units interspersed among the P units. The beginning of each segment is the start of a random access point, while each R unit provides a loss recovery point that can be placed independently of the I unit. This approach separates the random access point from the loss recovery points provided by the R units, and makes the stream more impervious to data losses without substantially impacting coding efficiency. The most important data units are transmitted with the most reliability to ensure that the majority of the data received by the client is usable. The I units are the least sensitive to transmission losses because they are coded using only their own data. While they provide the best coding efficiency, the P units are the most sensitive to data loss because the loss of one P unit renders useless all of the P units that depend on it. The remotely predicted units are dependent on the I unit, or in an alternative implementation, on another R unit.

## Claims

### Claim at position 1

-26. (canceled)

### Claim 27

A method of decoding video using a video decoder implemented in a computer system, the method comprising:
at the video decoder, receiving encoded video in a data stream, wherein the encoded video includes an entropy coded macroblock type for a current macroblock of a current frame;
with the video decoder, decoding video using the encoded video, including:
decoding the entropy coded macroblock type of the current macroblock to determine its macroblock type from among plural macroblock types possible for the current macroblock, the plural macroblock types including:
an intra type that indicates the current macroblock is to be decoded using intra decoding;
a first motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a first reference frame that immediately precedes the current frame; and
a second motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a second reference frame different than the first reference frame; and
decoding the current macroblock according to the macroblock type of the current macroblock, including:
when the macroblock type for the current macroblock is the intra type, decoding the current macroblock in an intra decoding mode;
when the macroblock type for the current macroblock is the first motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the first reference frame, wherein the motion compensation for the first motion-compensated prediction type uses at most one motion vector and one reference frame per block; and
when the macroblock type for the current macroblock is the second motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the second reference frame, wherein the motion compensation for the second motion-compensated prediction type uses at most one motion vector and one reference frame per block.

### Claim 28 (depends on 27)

The method of claim 27 wherein:
a first buffer stores the first reference frame;
a second buffer stores the second reference frame; and
the second reference frame is a reconstructed version of a previous frame having at least some macroblocks decoded using motion compensation.

### Claim 29 (depends on 28)

The method of claim 28 wherein, during decoding of plural frames that include the current frame and the previous frame, the video decoder updates the first buffer and the second buffer depending on classification of the plural frames, respectively, including:
if a given frame of the plural frames is an intra frame, storing a reconstructed version of the given frame in the first buffer and in the second buffer;
if the given frame is classified as a type of frame from which subsequent macroblocks with the second motion-compensated prediction type are predicted, storing the reconstructed version of the given frame in the first buffer and in the second buffer; and
otherwise, storing the reconstructed version of the given frame in the first buffer but not in the second buffer.

### Claim 30 (depends on 27)

The method of claim 27 further comprising:
storing a reconstructed version of the current frame in a first buffer used in motion-compensated prediction for subsequent macroblocks with the first motion-compensated prediction type; and
determining if the current frame is to be used as a reference frame for subsequent macroblocks with the second motion-compensated prediction type and, if so, storing the reconstructed version of the current frame in a second buffer.

### Claim 31 (depends on 27)

The method of claim 27 wherein the decoding the current macroblock further comprises, when the macroblock type for the current macroblock is the first motion-compensated prediction type or the second motion-compensated prediction type:
decoding error signal values for the current macroblock; and
combining the error signal values for the current macroblock with results of the motion compensation performed for the current macroblock.

### Claim 32 (depends on 31)

The method of claim 31 wherein:
the decoding of the entropy coded macroblock type for the current macroblock uses arithmetic decoding;
the decoding the current macroblock further comprises determining a motion vector for the current macroblock based at least in part upon arithmetic decoding of motion vector information for the current macroblock; and
the decoding the error signal values for the current macroblock includes:
arithmetic decoding of encoded transform coefficient values;
inverse quantization of the decoded transform coefficient values; and
performing an inverse DCT of the inverse quantized transform coefficient values.

### Claim 33 (depends on 27)

The method of claim 27 wherein use of the first motion-compensated prediction type promotes coding efficiency at the expense of loss recovery, and wherein use of the second motion-compensated prediction type promotes loss recovery at the expense of coding efficiency.

### Claim 34 (depends on 27)

The method of claim 27 wherein the first motion-compensated prediction type and the second motion-compensated prediction type are used for decoding of different objects in the first reference frame and second reference frame, respectively.

### Claim 35 (depends on 27)

The method of claim 27 wherein the current frame has a frame type that permits only intra decoding or inter decoding with forward motion-compensated prediction.

### Claim 36 (depends on 27)

The method of claim 27 wherein the video decoder is implemented using any one or more of:
software that executes on a general-purpose processor of the computer system, the software being stored in memory of the computer system;
special-purpose digital logic in the computer system; and
software the controls special-purpose digital logic in the computer system.

### Claim 37 (depends on 27)

The method of claim 27 wherein the computer system is any one of a desktop computer, hand-held device, other consumer electronic device, or distributed computing system.

### Claim 38

A method of encoding video using a video encoder implemented in a computer system, the method comprising:
with the video encoder, encoding video to produce encoded video for output to a bit stream, including:
selecting a macroblock type of a current macroblock of a current frame from among plural macroblock types possible for the current macroblock, the plural macroblock types including:
an intra type that indicates the current macroblock is encoded using intra coding;
a first motion-compensated prediction type that indicates motion compensation for the current macroblock is performed relative to a first reference frame that immediately precedes the current frame; and
a second motion-compensated prediction type that indicates motion compensation for the current macroblock is performed relative to a second reference frame different than the first reference frame;
entropy coding the macroblock type of the current macroblock; and
encoding the current macroblock according to the macroblock type of the current macroblock, including:
when the macroblock type for the current macroblock is the intra type, encoding the current macroblock in an intra coding mode;
when the macroblock type for the current macroblock is the first motion-compensated prediction type, encoding the current macroblock using motion compensation relative to the first reference frame, wherein the motion compensation for the first motion-compensated prediction type uses at most one motion vector and one reference frame per block; and
when the macroblock type for the current macroblock is the second motion-compensated prediction type, encoding the current macroblock using motion compensation relative to the second reference frame, wherein the motion compensation for the second motion-compensated prediction type uses at most one motion vector and one reference frame per block.
with the video encoder, outputting the encoded video in a data stream, wherein the encoded video includes the entropy coded macroblock type for the current macroblock of a current frame.

### Claim 39 (depends on 38)

The method of claim 38 wherein:
a first buffer stores the first reference frame;
a second buffer stores the second reference frame; and
the second reference frame is a reconstructed version of a previous frame having at least some macroblocks encoded and decoded using motion compensation.

### Claim 40 (depends on 39)

The method of claim 39 wherein, during encoding of plural frames that include the current frame and the previous frame, the video encoder updates the first buffer and the second buffer depending on classification of the plural frames, respectively, including:
if a given frame of the plural frames is an intra frame, storing a reconstructed version of the given frame in the first buffer and in the second buffer;
if the given frame is classified as a type of frame from which subsequent macroblocks with the second motion-compensated prediction type are predicted, storing the reconstructed version of the given frame in the first buffer and in the second buffer; and
otherwise, storing the reconstructed version of the given frame in the first buffer but not in the second buffer.

### Claim 41 (depends on 38)

The method of claim 38 wherein the encoding the current macroblock further comprises, when the macroblock type for the current macroblock is the first motion-compensated prediction type or the second motion-compensated prediction type:
determining error signal values for the current macroblock; and
encoding the error signal values for the current macroblock.

### Claim 42 (depends on 41)

The method of claim 41 wherein:
the entropy coding of the macroblock type for the current macroblock uses arithmetic coding;
the encoding the current macroblock further comprises encoding motion vector information for a motion vector for the current macroblock using arithmetic coding; and
the encoding the error signal values for the current macroblock includes arithmetic coding of quantized transform coefficient values;

### Claim 43 (depends on 38)

The method of claim 38 wherein the video encoder selects between the first and second motion-compensated prediction types for the current macroblock depending at least in part on a motion-compensated prediction error threshold.

### Claim 44

A computer-readable medium storing in non-volatile storage computer-executable instructions for causing a video decoder programmed thereby to perform a method comprising:
receiving encoded video in a data stream, wherein the encoded video includes an entropy coded macroblock type for a current macroblock of a current frame;
decoding video using the encoded video, including:
decoding the entropy coded macroblock type of the current macroblock to determine its macroblock type from among plural macroblock types possible for the current macroblock, the plural macroblock types including:
an intra type that indicates the current macroblock is to be decoded using intra decoding;
a first motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a first reference frame that immediately precedes the current frame; and
a second motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a second reference frame different than the first reference frame; and
decoding the current macroblock according to the macroblock type of the current macroblock, including:
when the macroblock type for the current macroblock is the intra type, decoding the current macroblock in an intra decoding mode;
when the macroblock type for the current macroblock is the first motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the first reference frame, wherein the motion compensation for the first motion-compensated prediction type uses at most one motion vector and one reference frame per block; and
when the macroblock type for the current macroblock is the second motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the second reference frame, wherein the motion compensation for the second motion-compensated prediction type uses at most one motion vector and one reference frame per block.

### Claim 45 (depends on 44)

The computer-readable medium of claim 44 wherein the method further comprises:
storing a reconstructed version of the current frame in a first buffer used in motion-compensated prediction for subsequent macroblocks with the first motion-compensated prediction type; and
determining if the current frame is to be used as a reference frame for subsequent macroblocks with the second motion-compensated prediction type and, if so, storing the reconstructed version of the current frame in a second buffer.

### Claim 46 (depends on 44)

The computer-readable medium of claim 44 wherein:
the decoding of the entropy coded macroblock type for the current macroblock uses arithmetic decoding;
the decoding the current macroblock further comprises determining a motion vector for the current macroblock based at least in part upon arithmetic decoding of motion vector information for the current macroblock.
