# US 2020/0007869 A1 · Scalable video coding using inter-layer prediction contribution to enhancement layer prediction

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0007869 A1
- Document ID: US20200007869A1
- Status: Published application
- Kind: A1
- Application number: 16/459,940
- Filed: 2019-07-02
- Published: 2020-01-02
- Earliest claimed: 2012-10-01
- Assignees: Ge Video Compression, LLC
- Applicants: GE Video Compression, LLC
- Inventors: Tobias HINZ; Haricharan LAKSHMAN; Jan STEGEMANN; Philipp HELLE; Mischa SIEKMANN; Karsten SUEHRING; Detlev MARPE; Heiko SCHWARZ; Christian BARTNIK; Ali Atef Ibrahim KHAIRAT ABDELHAMID; Heiner KIRCHHOFFER; Thomas WIEGAND
- Web page: https://patentgrep.com/patent/US20200007869A1

## CPC

- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/107: between spatial and temporal predictive coding, e.g. picture refresh
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/119: Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference
- H04N 19/157: Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
- H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/18: the unit being a set of transform coefficients
- H04N 19/187: the unit being a scalable video layer
- H04N 19/198: {including smoothing of a sequence of encoding parameters, e.g. by averaging, by choice of the maximum, minimum or median value}
- H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence)
- H04N 19/33: in the spatial domain
- H04N 19/48: using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data (motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513)
- H04N 19/503: involving temporal prediction (adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109)
- H04N 19/52: by predictive encoding
- H04N 19/587: involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
- H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
- H04N 19/593: involving spatial prediction techniques
- H04N 19/61: in combination with predictive coding
- H04N 19/615: using motion compensated temporal filtering [MCTF]
- H04N 19/82: involving filtering within a prediction loop
- H04N 19/865: {with detection of the former encoding block subdivision in decompressed video}
- H04N 19/11: among a plurality of spatial predictive coding modes
- H04N 19/34: Scalability techniques involving progressive bit-plane based encoding of the enhancement layer, e.g. fine granular scalability [FGS]
- H04N 19/36: Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
- H04N 19/463: by compressing encoding parameters before transmission
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A scalable video decoder is described which is configured to reconstruct a base layer signal from a coded data stream to obtain a reconstructed base layer signal; and reconstruct an enhancement layer signal including spatially or temporally predicting a portion of an enhancement layer signal, currently to be reconstructed, from an already reconstructed portion of the enhancement layer signal to obtain an enhancement layer internal prediction signal; forming, at the portion currently to be reconstructed, a weighted average of an inter-layer prediction signal obtained from the reconstructed base layer signal, and the enhancement layer internal prediction signal to obtain an enhancement layer prediction signal such that a weighting between the inter-layer prediction signal and the enhancement layer internal prediction signal varies over different spatial frequency components; and predictively reconstructing the enhancement layer signal using the enhancement layer prediction signal.

## Claims

### Claim 1

A video decoder comprising:
a first coding unit configured for reconstructing, using a processor, a base layer signal from a coded data stream to obtain a reconstructed base layer signal; and
a second coding unit configured for reconstructing, using a processor, an enhancement layer signal, the reconstructing comprising:
predicting, based on a reconstructed portion of the enhancement layer signal, a current portion of the enhancement layer signal to obtain an enhancement layer internal signal,
obtaining an inter-layer prediction signal associated with the current portion of the enhancement layer signal based on the reconstructed base layer signal,
extracting, from the coded data stream, weighting parameters for the inter-layer prediction signal and the enhancement layer internal signal,
determining a weighted average of the inter-layer prediction signal and the enhancement layer internal signal to obtain an enhancement layer prediction signal by:
filtering the inter-layer prediction signal with a first filter to generate a filtered inter-layer prediction signal,
filtering the enhancement layer internal signal with a second filter to generate a filtered enhancement layer internal signal,
applying the weighting parameters to the filtered inter-layer prediction signal and the filtered enhancement layer internal signal, respectively, to obtain the enhancement layer prediction signal, and
predictively reconstructing the enhancement layer signal based on the enhancement layer prediction signal.

### Claim 2 (depends on 1)

The video decoder according to claim 1, wherein obtaining the inter-layer prediction signal comprises performing a resolution refinement to obtain the inter-layer prediction signal.

### Claim 3 (depends on 1)

The video decoder according to claim 1, wherein the weighting parameters for the inter-layer prediction signal and the enhancement layer internal signal vary over different spatial frequency components.

### Claim 4 (depends on 1)

The video decoder according to claim 1, wherein the first filter is a low-pass filter and the second filter is a high-pass filter.

### Claim 5 (depends on 1)

The video decoder according to claim 1, wherein the first and second filters form a quadrature mirror filter pair.

### Claim 6 (depends on 1)

The video decoder according to claim 1, wherein determining comprises the weighted average transforming the inter-layer prediction signal and the enhancement layer internal signal to obtain transform coefficients, and superimposing the transform coefficients using weighting parameters of different ratios therebetween for different spatial frequency components to obtain superimposed transform coefficients.

### Claim 7 (depends on 6)

The video decoder according to claim 6, wherein determining further comprises inverse transforming the superimposed transform coefficients to obtain the enhancement layer prediction signal.

### Claim 8 (depends on 6)

The video decoder according to claim 6, wherein predictively reconstructing the enhancement layer signal using the enhancement layer prediction signal comprises extracting transform coefficient levels for the enhancement layer signal from a coded data stream, performing a sum of the transform coefficient levels and the superimposed transform coefficients to obtain a transformed version of the enhancement layer signal, and performing an inverse transform of the transformed version of the enhancement layer signal to reconstruct the enhancement layer signal.

### Claim 9 (depends on 1)

The video decoder according to claim 1, wherein, for each spatial frequency component, a sum of the weighting parameters for the inter-layer prediction signal and the enhancement layer internal signal equal a value that is same for all spatial frequency components.

### Claim 10 (depends on 1)

The video decoder according to claim 1, wherein the weighting parameter for weighting the inter-layer prediction signal correspond to a low-pass transfer function, and the weighting parameter for weighting the enhancement layer internal signal correspond to a high-pass transfer function.

### Claim 11 (depends on 1)

The video decoder according to claim 1, wherein reconstructing the enhancement layer signal comprises
computing a difference signal between an already reconstructed portion of an enhancement layer signal and the inter-layer prediction signal,
spatially predicting the difference signal at a first portion collocated to the current portion of the enhancement layer signal from a second portion of the difference signal,
spatially neighboring the first portion and belonging to the already reconstructed portion of the enhancement layer signal to obtain a spatial intra prediction signal, and
combining the inter-layer prediction signal and the spatial intra prediction signal to reconstruct the enhancement layer prediction signal.

### Claim 12 (depends on 1)

The video decoder according to claim 1, wherein reconstructing the enhancement layer signal comprises
computing a difference signal between an already reconstructed portion of an enhancement layer signal and the inter-layer prediction signal,
temporally predicting the difference signal at a first portion collocated to the current portion of the enhancement layer signal from a second portion of the difference signal, belonging to previously reconstructed frames of the enhancement layer signal to obtain a temporally predicted difference signal, and
combining the inter-layer prediction signal and the temporally predicted difference signal to reconstruct the enhancement layer prediction signal.

### Claim 13

A video encoder comprising:
a first encoding unit configured encoding, using a processor, a base layer signal into a coded data stream in a manner allowing reconstructing a reconstructed base layer signal from the coded data stream; and
a second encoding unit configured for encoding, using a processor, an enhancement layer signal, the encoding the enhancement layer signal comprising:
predicting a current portion of an enhancement layer from an encoded portion of the enhancement layer signal to obtain an enhancement layer internal signal,
obtaining an inter-layer prediction signal associated with the current portion of the enhancement layer signal based on the reconstructed base layer signal,
inserting, into the coded data stream, weighting parameters for the inter-layer prediction signal and the enhancement layer internal signal,
determining a weighted average of the inter-layer prediction signal and the enhancement layer internal signal to obtain an enhancement layer prediction signal by:
filtering the inter-layer prediction signal with a first filter to generate a filtered inter-layer prediction signal, and filtering the enhancement layer internal signal with a second filter to generate a filtered enhancement layer internal signal, and
applying the weighting parameters to the filtered inter-layer prediction signal and the filtered enhancement layer internal signal to obtain the enhancement layer prediction signal, and
predictively encoding the enhancement layer signal based on the enhancement layer prediction signal.

### Claim 14 (depends on 13)

The video encoder according to claim 13, wherein the weighting parameters for the inter-layer prediction signal and the enhancement layer internal signal vary over different spatial frequency components.

### Claim 15 (depends on 13)

The video encoder according to claim 13, wherein obtaining the inter-layer prediction signal comprises performing a resolution refinement to obtain the inter-layer prediction signal.

### Claim 16 (depends on 13)

The video encoder according to claim 13, wherein the weighting parameters for the inter-layer prediction signal and the enhancement layer internal signal vary over different spatial frequency components.

### Claim 17 (depends on 13)

The video encoder according to claim 13, wherein the first filter is a low-pass filter and the second filter is a high-pass filter.

### Claim 18 (depends on 13)

The video encoder according to claim 13, wherein the first and second filters form a quadrature mirror filter pair.

### Claim 19 (depends on 13)

The video encoder according to claim 13, wherein, for each spatial frequency component, a sum of the weighting parameters for the inter-layer prediction signal and the enhancement layer internal signal equal a value that is same for all spatial frequency components.

### Claim 20

A non-transitory computer readable medium including a computer program comprising a program code for performing, when running on a computer, a method including:
reconstructing a base layer signal from a coded data stream to obtain a reconstructed base layer signal; and
reconstructing an enhancement layer signal, the reconstructing comprising:
predicting, based on a reconstructed portion of the enhancement layer signal, a current portion of the enhancement layer signal to obtain an enhancement layer internal signal,
obtaining an inter-layer prediction signal associated with the current portion of the enhancement layer signal based on the reconstructed base layer signal,
extracting, from the coded data stream, weighting parameters for the inter-layer prediction signal and the enhancement layer internal signal,
determining a weighted average of the inter-layer prediction signal and the enhancement layer internal signal to obtain an enhancement layer prediction signal by:
filtering the inter-layer prediction signal with a first filter to generate a filtered inter-layer prediction signal, and filtering the enhancement layer internal signal with a second filter to generate a filtered inter-layer prediction signal, and
applying the weighting parameters to the filtered inter-layer prediction signal and the filtered enhancement layer internal signal to obtain the enhancement layer prediction signal, and
predictively reconstructing the enhancement layer signal based on the enhancement layer prediction signal.
