# US 2021/0084313 A1 · Refined block-based predictive coding and decoding of a picture

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0084313 A1
- Document ID: US20210084313A1
- Status: Published application
- Kind: A1
- Application number: 17/036,828
- Filed: 2020-09-29
- Published: 2021-03-18
- Earliest claimed: 2018-03-29
- Assignees: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
- Applicants: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
- Inventors: Phan Hoang Tung NGUYEN; Gerhard TECH; Jonathan PFAFF; Michael SCHÄFER; Jennifer RASCH; Tobias HINZ; Heiko SCHWARZ; Detlev MARPE; Thomas WIEGAND
- Web page: https://patentgrep.com/patent/US20210084313A1

## CPC

- H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks
- H04N 19/82: involving filtering within a prediction loop
- G06T 5/10: using non-spatial domain filtering
- G06T 5/70: Denoising; Smoothing
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/176: the region being a block, e.g. a macroblock
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20052: Discrete cosine transform [DCT]
- G06T 2207/20056: Discrete and fast Fourier transform, [DFT, FFT]
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

The refinement of block-based predictive coding and decoding of pictures as applicable in, e.g., hybrid video codecs, using additional reconstructed samples or residuals is described. An apparatus for block-based predictive decoding of a picture is described. The apparatus includes a combiner configured to combine a residual signal a predetermined block of the picture and a reference signal for the predetermined block so as to obtain a first spectrum, the residual signal correcting a prediction error of a prediction of the predetermined block of the picture; a reducer configured to perform thresholding on the first spectrum to obtain a second spectrum so that coefficients below a threshold value are set to a predefined value; an extractor configured to obtain from the second spectrum a modified version of the residual signal; and a reconstructor block configured to decode the predetermined block of the picture from the data stream on the basis of the modified version of the residual signal.

## Claims

### Claim 1

An apparatus for block-based predictive decoding of a picture, comprising:
a combiner configured to combine a residual signal of a predetermined block of the picture and a reference signal for the predetermined block so as to obtain a first set of coefficients, the residual signal correcting a prediction error of a prediction of the predetermined block of the picture;
a reducer configured to perform thresholding on the first set of coefficients to obtain a second set of coefficients so that coefficients below a threshold value are set to a predefined value;
an extractor configured to obtain from the second set of coefficients a modified version of the residual signal; and
a reconstructor block configured to decode the predetermined block of the picture from the data stream on the basis of the modified version of the residual signal.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the reference signal comprises previous residual signals of a neighborhood of the predetermined block.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the predetermined block of the picture is an inter-predicted block.

### Claim 4

An apparatus for block-based predictive decoding of a picture, comprising:
an decoding block configured to decode a residual signal from a data stream, the residual signal correcting a prediction error of a prediction signal for a predetermined block of the picture; and
a comparator configured to
compare, for each coefficient position within a reference signal for the predetermined block, a coefficient against a threshold value, and
if the coefficient value is below the threshold, set, at the corresponding coefficient position within an employed signal, the coefficient to a predefined value, thereby obtaining a modified employed signal, wherein the employed signal comprises the prediction signal and/or the residual signal;
the decoding block configured to decode the predetermined block from the data stream on the basis of the modified employed signal.

### Claim 5 (depends on 4)

The apparatus of claim 4, comprising:
a combiner configured to combine the employed signal and the reference so as to obtain a first set of coefficients, wherein the comparator is configured to set, at the corresponding coefficient position within the first set of coefficients, the coefficient to the predefined value, thereby obtaining a modified first set of coefficients, and
an extractor configured to obtain from the modified first set of coefficients the modified employed signal.

### Claim 6 (depends on 5)

The apparatus of claim 5, comprising:
a decomposer configured to perform a transform on a region composed of the prediction signal and the reference signal so as to obtain a first set of coefficients of the region; and
a composer configured to subject the modified first set of coefficients to a re-transform so as to obtain a modified version of the region including the modified prediction signal.

### Claim 7

An apparatus for block-based predictive decoding of a picture, comprising:
a reconstructor block configured to decode a predetermined block of a picture from a data stream on the basis of a residual signal, the residual signal correcting a prediction error of a prediction of a predetermined block of the picture;
a prediction provider configured to provide the prediction of the predetermined block of the picture;
a first refinement block for a refinement of the prediction of the predetermined block of the picture; and
a second refinement block for a refinement of the residual signal.

### Claim at position 8

8-12. (canceled)

### Claim 13

An apparatus for block-based predictive decoding of a picture, comprising:
a prediction provider configured to predict a predetermined block of the picture to obtain a first version of a prediction signal for the predetermined block;
a decomposer configured to perform a transform a region composed of the first version of the prediction signal and a reference signal for the predetermined block so as to obtain a first set of coefficients of the region;
a reducer configured to perform thresholding on the first set of coefficients to obtain a second set of coefficients so that coefficients below a threshold value are set to a predefined value;
a composer configured to subject the second set of coefficients to a re-transform so as to directly obtain a second version of the prediction signal; and
a reconstructor block configured to decode the predetermined block of the picture from the data stream on the basis of the second version of the prediction signal.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the second version of the prediction signal is directly obtained without a full reconstruction of the region in the spatial domain.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein the second version of the prediction signal is directly obtained by a partial reconstruction of the region in the spatial domain.

### Claim 16

(canceled)

### Claim 17 (depends on 1)

The apparatus of claim 1, wherein the reference signal comprises a previously decoded version of a neighborhood of the predetermined block.

### Claim at position 14

18-35. (canceled)

### Claim 36 (depends on 1)

The apparatus of claim 1, wherein setting the coefficients by the thresholding to the predefined value includes:
setting a coefficient to a certain value that depends on the threshold value, e.g., a preset fraction of the threshold value, like half the threshold, or a preset absolute value, like zero, and/or
reducing a coefficient by a certain amount, e.g., by moving a value of the coefficient towards a preset value, like zero, e.g., by an amount equal to the threshold value or to a fraction of the threshold, like half the threshold.

### Claim 37 (depends on 1)

The apparatus of claim 1, wherein setting a coefficient to the predefined value comprises quantizing the coefficient associated with the predefined value.

### Claim at position 17

38-47. (canceled)

### Claim 48 (depends on 1)

The apparatus of claim 1, wherein the neighborhood of the predetermined block comprises:
a spatial area surrounding one of more sides of the predetermined block, and/or
a spatial area offset relative to the predetermined block, and/or
a spatial area surrounding one of more sides of a reference block.

### Claim 49 (depends on 48)

The apparatus of claim 48, wherein the spatial area surrounding one of more sides of the predetermined block comprises an extension of the predetermined block to the left so that the total width is N and to the top so that the total height is M, both N and M being advantageously equal.

### Claim at position 20

50-56. (canceled)

### Claim 57 (depends on 1)

The apparatus of claim 1, wherein the apparatus is configured to transmit side information describing a configuration of the encoding process, e.g., a location of the reference signal, a threshold value, an index indicating a threshold value and the like, for each partitioning structure separately or for several partitioning structures.

### Claim 58 (depends on 57)

The apparatus of claim 57, wherein, when transmitting the configuration of the encoding process for several partitioning structures, the apparatus is configured to apply an iterative strategy including:
selecting parameters for the encoding process based of a fixed grid containing several partitioning structures, and
recalculating the grid using the results of the encoding process using the selected parameters.

### Claim 59 (depends on 1)

The apparatus of claim 1, wherein
the set of coefficients comprises a spectrum, and/or
the coefficient positions comprise spectral positions, and/or
the coefficients comprise spectral coefficients, and/or
the composer comprises a spectral composer, and/or
the decomposer comprises a spectral decomposer, and/or
the transform comprises a base transform, e.g., a spectral transform and/or a change of basis, like a Singular Value Decomposition (SVC), a FFT or a DCT, and/or the re-transform comprises an inverse base transform, e.g., an inverse spectral base transform.

### Claim 60

A method for block-based predictive decoding of a picture, comprising:
combining a residual signal a predetermined block of the picture and a reference signal for the predetermined block so as to obtain a first set of coefficients, the residual signal correcting a prediction error of a prediction of the predetermined block of the picture;
performing thresholding on the first set of coefficients to obtain a second set of coefficients so that coefficients below a threshold value are set to a predefined value;
obtaining from the second set of coefficients a modified version of the residual signal; and
decoding the predetermined block of the picture from the data stream on the basis of the modified version of the residual signal.

### Claim 61

A method for block-based predictive decoding of a picture, comprising:
decoding a residual signal from a data stream, the residual signal correcting a prediction error of a prediction signal for a predetermined block of the picture; and
comparing, for each coefficient position within a reference signal for the predetermined block, a coefficient against a threshold value; and
if the coefficient value is below the threshold, setting, at the corresponding coefficient position within an employed signal, the coefficient to the predefined value, thereby
obtaining a modified employed signal, wherein the employed signal comprises the prediction signal and/or the residual signal;
wherein the decoding includes decoding the predetermined block from the data stream on the basis of the modified employed signal.

### Claim at position 26

62-66. (canceled)

### Claim 67

A method for block-based predictive decoding of a picture, comprising:
predicting a predetermined block of the picture to obtain a first version of a prediction signal for the predetermined block;
performing a transform on a region composed of the first version of the prediction signal and a reference signal for the predetermined block so as to obtain a first set of coefficients of the region;
performing thresholding on the first set of coefficients to obtain a second set of coefficients so that coefficients below a threshold value are set to a predefined value;
subjecting the second set of coefficients to a re-transform so as to directly obtain a second version of the prediction signal; and
decoding the predetermined block of the picture from the data stream on the basis of the second version of the prediction signal.

### Claim at position 28

68-77. (canceled)

### Claim 78 (depends on 60)

A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 60.

### Claim 79 (depends on 60)

A data stream having a picture encoded thereinto, the data stream being generated by a method according to claim 60.
