# US 2012/0250759 A1 · Apparatus and Method for Generating a Coded Video Sequence and for Decoding a Coded Video Sequence by Using an Intermediate Layer Residual Value Prediction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2012/0250759 A1
- Document ID: US20120250759A1
- Status: Published application
- Kind: A1
- Application number: 13/523,975
- Filed: 2012-06-15
- Published: 2012-10-04
- Earliest claimed: 2004-10-15
- Assignees: Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V.
- Inventors: Heiko Schwarz; Detlev Marpe; Thomas Wiegand
- Web page: https://patentgrep.com/patent/US20120250759A1

## CPC

- H04N 19/56: Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
- H04N 19/103: Selection of coding mode or of prediction mode
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/107: between spatial and temporal predictive coding, e.g. picture refresh
- H04N 19/109: among a plurality of temporal predictive coding modes
- H04N 19/11: among a plurality of spatial predictive coding modes
- H04N 19/147: according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567)
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/19: using optimisation based on Lagrange multipliers
- H04N 19/31: in the temporal domain
- H04N 19/33: in the spatial domain
- H04N 19/36: Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/51: Motion estimation or motion compensation
- H04N 19/52: by predictive encoding
- H04N 19/567: Motion estimation based on rate distortion criteria
- H04N 19/61: in combination with predictive coding
- H04N 19/615: using motion compensated temporal filtering [MCTF]
- H04N 19/63: using sub-band based transform, e.g. wavelets
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N 19/13: Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
- H04N 19/42: characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A video coder performs a motion-compensated prediction both in the base layer and in an enhancement layer to determine motion data of the enhancement layer by using the motion data from the base layer and/or to predict sequences of residual error pictures after the motion-compensated prediction in the enhancement layer by using sequences of residual error pictures from the base layer via an intermediate layer predictor. On the decoder side, an intermediate layer combiner is used for canceling this intermediate layer prediction. Thereby, the data rate is improved compared to scalability schemes without intermediate layer prediction with the same picture quality.

## Claims

### Claim 1

An apparatus for generating a coded video sequence having a base scaling layer and an enhancement scaling layer, comprising:
a base motion compensator for calculating base motion data, which indicate how a macroblock in a current picture has moved in relation to another picture in a group of pictures;
a base motion predictor for calculating a base sequence of residual error pictures by using the base motion data;
a base picture coder formed to calculate the coded base scaling layer from the base sequence of residual error pictures;
an enhancement motion compensator for determining enhancement motion data;
an enhancement motion predictor for calculating an enhancement sequence of residual error pictures;
an intermediate layer predictor for calculating enhancement prediction residual error pictures by using the enhancement sequence of residual error pictures and by using information about the base sequence of residual error pictures; and
an enhancement picture coder for coding the enhancement prediction residual error pictures to obtain the coded enhancement scaling layer.

### Claim 2 (depends on 1)

The apparatus according to claim 1, wherein the base picture coder is formed to perform a quantization with a base quantization parameter;
wherein the enhancement picture coder is formed to perform a quantization with an enhancement quantization parameter, wherein the enhancement quantization parameter can result in a finer quantization than the base quantization parameter;
wherein the base picture coder is formed to reconstruct the base sequence of residual error pictures quantized with the first quantization parameter to obtain a reconstructed base sequence and
wherein the intermediate layer predictor is formed to calculate the enhancement prediction residual error pictures by using the enhancement sequence of residual error pictures and the reconstructed base sequence of residual error pictures as information about the base sequence of residual error pictures.

### Claim 3 (depends on 1)

The apparatus according to claim 1, further comprising:
a decimator for decimating a resolution of the group of pictures, wherein the decimator is formed to provide a group of pictures with a base resolution to the base compensator, which is smaller than an enhancement resolution of a group of pictures which is provided to the enhancement motion compensator; and
an interpolator for spatially interpolating the base sequence of residual error pictures or a reconstructed base sequence of residual error pictures to obtain an interpolated base sequence of residual error pictures, which can be supplied to the intermediate layer predictor as information about the base sequence of residual error pictures.

### Claim 4 (depends on 3)

The apparatus according to claim 3, further comprising a motion data upsampler to adapt the base motion data to the enhancement resolution.

### Claim 5 (depends on 1)

The apparatus according to claim 1, wherein the base motion predictor is formed to subtract a macroblock predicted by using the base motion data from a current macroblock to obtain a macroblock of a residual error picture of the base sequence of residual error pictures.

### Claim 6 (depends on 1)

The apparatus according to claim 1, wherein the intermediate layer predictor is formed to decide adaptively for blocks, whether a coder gain can be increased by applying a prediction in comparison to a usage of a block from the enhancement sequence of residual error pictures.

### Claim 7 (depends on 1)

The apparatus according to claim 1, wherein the intermediate layer predictor is formed to subtract a macroblock predicted by using a macroblock from the base sequence of residual error pictures or the reconstructed base sequence of residual error pictures or the interpolated base sequence of residual error pictures or the reconstructed and interpolated base sequence of residual error pictures from a current macroblock of the enhancement sequence of residual error pictures to obtain a macroblock of an enhancement prediction residual error picture.

### Claim 8 (depends on 1)

The apparatus according to claim 1, wherein the enhancement motion compensator is formed to use the base motion data or scaled motion data in dependence on a mode control signal.

### Claim 9 (depends on 1)

The apparatus according to claim 1, wherein the base scaling layer has a lower resolution than the enhancement scaling layer, and wherein the enhancement motion compensator is formed to determine residual motion data in dependence on a control signal, and wherein the enhancement picture coder is formed to code the residual motion data into the enhancement scaling layer.

### Claim 10

A method for generating a coded video sequence having a base scaling layer and an enhancement scaling layer, comprising:
calculating base motion data, which indicate how a macroblock in a current picture has moved in relation to another picture in a group of pictures;
calculating a base sequence of residual error pictures by using the base motion data;
coding information about the base sequence of residual error pictures to calculate the coded base scaling layer from the base sequence of residual error pictures;
determining enhancement motion data;
calculating an enhancement sequence of residual error pictures;
calculating enhancement prediction residual error pictures by using the enhancement sequence of residual error pictures and by using information about the base sequence of residual error pictures; and
coding the enhancement prediction residual error pictures to obtain the coded enhancement scaling layer.

### Claim 11

An apparatus for decoding a coded video sequence having a base scaling layer and an enhancement scaling layer, comprising:
a base picture decoder configured for decoding the base scaling layer to obtain a decoded base sequence of residual error pictures and base layer motion data, the base layer motion data indicating how a macroblock in a current picture has moved in relation to another picture in a group of pictures;
an enhancement picture decoder configured for decoding the enhancement scaling layer to obtain enhancement layer prediction residual error pictures;
an intermediate layer combiner configured for combining the decoded base sequence of residual error pictures or an interpolated base sequence of residual error pictures with the enhancement layer prediction residual error pictures to obtain an enhancement sequence of residual error pictures; and
an enhancement motion combiner configured for obtaining a sequence of pictures of the enhancement scaling layer by using the enhancement sequence of residual error pictures and enhancement layer motion data,
wherein the pictures of the base scaling layer have a lower resolution than the pictures of the enhancement scaling layer, and
wherein the apparatus further comprises a motion data upsampler to convert the base motion data to the enhancement resolution to obtain the enhancement motion data or a prediction of the enhancement motion data.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein the base picture decoder is formed to dequantize by using a base quantization parameter which is larger than an enhancement quantization parameter, and
wherein the intermediate layer combiner is formed to use the decoded base sequence of residual error pictures or an interpolated version of the same.

### Claim 13 (depends on 11)

The apparatus according to claim 11, wherein the coded video sequence has residual motion data, and
having a motion data determiner to combine the residual motion data with the prediction of the enhancement motion data to determine the enhancement motion data.

### Claim 14 (depends on 11)

The apparatus according to claim 11, wherein the base motion combiner and the enhancement motion combiner are formed to perform only an inverse prediction operation or an inverse prediction operation and an inverse update operation.

### Claim 15 (depends on 11)

The apparatus according to claim 11, wherein the coded video sequence has enhancement motion data for macroblocks, wherein a motion data control signal is associated to the enhancement motion data, which indicates whether a prediction signal is underlying the enhancement motion data, which is derived from a scaled motion vector of the base motion data, or whether a prediction signal is underlying the enhancement motion data, which is not derived from the base motion data, and wherein the apparatus is formed to read the motion data control signal and to calculate the enhancement motion data in dependence on the control signal by using the scaled version of the base motion data.

### Claim 16 (depends on 11)

The apparatus according to claim 11, wherein the coded video sequence for macroblocks has an intermediate layer prediction control signal in the enhancement scaling layer, which indicates whether a macroblock has been generated due to an intermediate layer prediction or without an intermediate layer prediction, and
wherein the apparatus is further formed to activate the intermediate layer combiner only when the intermediate layer prediction control signal indicates an intermediate layer prediction for a considered macroblock.

### Claim 17 (depends on 11)

Apparatus according to claim 11, further comprising a base motion combiner to obtain a sequence of pictures of the base scaling layer by using the base motion data and the decoded base sequence of residual error pictures, wherein the base motion combiner and the enhancement motion combiner are formed to perform an inverse prediction operation and an inverse update operation.

### Claim 18

A method for decoding a coded video sequence with a base scaling layer and an enhancement scaling layer, comprising:
decoding, by a base picture decoder, the base scaling layer to obtain a decoded base sequence of residual error pictures and base layer motion data, the base layer motion data indicating how a macroblock in a current picture has moved in relation to another picture in a group of pictures;
decoding, by an enhancement picture decoder, the enhancement scaling layer to obtain enhancement prediction layer residual error pictures;
combining, by an intermediate layer combiner, the decoded base sequence of residual error pictures or an interpolated base sequence of residual error pictures with the enhancement layer prediction residual error pictures to obtain an enhancement sequence of residual error pictures; and
performing, by an enhancement motion combiner, an enhancement motion combination to obtain a sequence of pictures of the enhancement scaling layer by using the enhancement sequence of residual error pictures and enhancement motion data,
wherein the pictures of the base scaling layer have a lower resolution than the pictures of the enhancement scaling layer, and
wherein the method further comprises converting the base motion data to the enhancement resolution to obtain the enhancement motion data or a prediction of the enhancement motion data.

### Claim 19

Non-transitory storage medium having stored thereon a computer-readable storage medium encoded with a computer program, causing a computer to execute a method for generating a coded video sequence having a base scaling layer and an enhancement scaling layer, comprising:
calculating base motion data, which indicate how a macroblock in a current picture has moved in relation to another picture in a group of pictures;
calculating a base sequence of residual error pictures by using the base motion data;
coding information about the base sequence of residual error pictures to calculate the coded base scaling layer from the base sequence of residual error pictures;
determining enhancement motion data;
calculating an enhancement sequence of residual error pictures;
calculating enhancement prediction residual error pictures by using the enhancement sequence of residual error pictures and by using information about the base sequence of residual error pictures; and
coding the enhancement prediction residual error pictures to obtain the coded enhancement scaling layer.

### Claim 20

Non-transitory storage medium having stored thereon a computer-readable storage medium encoded with a computer program, causing a computer to execute a method for decoding a coded video sequence with a base scaling layer and an enhancement scaling layer, the method comprising:
decoding the base scaling layer to obtain a decoded base sequence of residual error pictures and base layer motion data, the base layer motion data indicating how a macroblock in a current picture has moved in relation to another picture in a group of pictures;
decoding the enhancement scaling layer to obtain enhancement layer prediction residual error pictures;
combining the decoded base sequence of residual error pictures or an interpolated base sequence of residual error pictures with the enhancement layer prediction residual error pictures to obtain an enhancement sequence of residual error pictures; and
performing an enhancement motion combination to obtain a sequence of pictures of the enhancement scaling layer by using the enhancement sequence of residual error pictures and enhancement motion data,
wherein the pictures of the base scaling layer have a lower resolution than the pictures of the enhancement scaling layer, and
wherein the method further comprises converting the base motion data to the enhancement resolution to obtain the enhancement motion data or a prediction of the enhancement motion data.
