# US 11,115,682 B2 · Block-based predictive coding and decoding of a picture > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,115,682 B2 - Document ID: US11115682B2 - Status: Granted - Kind: B2 - Application number: 16/503,004 - Filed: 2019-07-03 - Published: 2019-10-24 - Granted: 2021-09-07 - Earliest claimed: 2017-01-05 - Assignees: Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V. - Applicants: Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V. - Inventors: Jonathan Pfaff; Martin Winken; Christian Rudat; Heiko Schwarz; Thomas Wiegand; Detlev Marpe - Web page: https://patentgrep.com/patent/US11115682B2 ## CPC - H04N 19/65: using error resilience - H04N 19/82: involving filtering within a prediction loop - H04L 65/70: Media network packetisation - H04L 65/80: Responding to QoS - H04N 19/13: Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC] - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/48: using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data (motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513) - H04N 19/60: using transform coding - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A previously encoded or reconstructed version of a neighborhood of a predetermined block to be predicted is exploited so as to result into a more efficient predictive coding of the prediction block. In particular, a spectral decomposition of a region composed of this neighborhood and a first version of a predicted filling of the predetermined block results into a first spectrum which is subject to noise reduction and the thus resulting second spectrum may be subject to a spectral composition, thereby resulting in a modified version of this region including a second version of the predicted filling of the predetermined block. Owing to the exploitation of the already processed, i.e. encoded/reconstructed, neighborhood of the predetermined block, the second version of the predicted filling of the predetermined block tends to improve the coding efficiency. ## Claims ### Claim 1 An apparatus for block-based predictive decoding of a picture, comprising: a prediction provider configured to predict a predetermined block of the picture to acquire a first version of a predicted filling of the predetermined block; a spectral decomposer configured to spectrally decompose a region composed of the first version of the predicted filling of the predetermined block and a reconstructed version of a neighborhood of the predetermined block so as to acquire a first spectrum of the region; a noise reducer configured to perform noise reduction on the first spectrum to acquire a second spectrum; a spectral composer configured to subject the second spectrum to spectral composition so as to acquire a modified version of the region including a second version of the predicted filling of the predetermined block; a reconstructor configured to decode a reconstructed version of the predetermined block from a data stream on the basis of the second version of the predicted filling, wherein the reconstructor is configured to decode a first residual signal from the data stream and correct a prediction signal for the neighborhood using the first residual signal to obtain the reconstructed version of the neighborhood of the predetermined block, and decode a second residual signal from the data stream and correct the second version of the predicted filling of the predetermined block using the second residual signal to obtain the reconstructed version of the predetermined block. ### Claim 2 (depends on 1) The apparatus according to claim 1, wherein the prediction provider is configured to predict the predetermined block using spatial and/or temporal prediction. ### Claim 3 (depends on 1) The apparatus according to claim 1, wherein the spectral decomposer is configured such that the neighborhood of the predetermined block is a spatial extension of the predetermined block beyond a contiguous half of a circumference of the predetermined block. ### Claim 4 (depends on 1) The apparatus according to claim 1, wherein the predetermined block is rectangular and the circumference of the predetermined block comprises a first, second, third and fourth side sequentially arranged along the circumference of the predetermined block clock-wise, wherein the contiguous half comprises the first and fourth sides. ### Claim 5 (depends on 4) The apparatus according to claim 4, wherein the region is rectangular and the circumference of the region includes the second and third sides of the predetermined block. ### Claim 6 (depends on 1) The apparatus according to claim 1, wherein the spectral decomposer and the spectral composer are configured such that the spectral decomposition and the spectral composition are mutually inverse with the spectral decomposition being a DST, DCT or wavelet transform. ### Claim 7 (depends on 1) The apparatus according to claim 1, wherein the noise reducer is configured to subject the first spectrum to thresholding. ### Claim 8 (depends on 1) The apparatus according to claim 1, wherein the reconstructor is configured to decode a residual signal from the data stream and correct the second version of the predicted filling of the predetermined block using the residual signal. ### Claim 9 (depends on 8) The apparatus according to claim 8, wherein the reconstructor is configured to decode the residual signal from the data stream in a spectral domain, transform the residual signal into spatial domain and perform the correction in the spatial domain. ### Claim 10 (depends on 9) The apparatus according to claim 9, wherein the reconstructor is configured to perform the transforming of the residual signal using a transform type also used by the spectral composer for the spectral composition. ### Claim 11 (depends on 1) The apparatus according to claim 1, wherein the reconstructor is configured to switch between using the first version of the predicted filling and the second version of the predicted filling depending on a first signalization in the data stream. ### Claim 12 (depends on 11) The apparatus according to claim 11, wherein the apparatus is configured to decode the first signalization from the data stream at sub-picture granularity. ### Claim 13 (depends on 11) The apparatus according to claim 11, wherein the apparatus is configured to decode the first signalization from the data stream using spatial prediction and/or using entropy decoding using a spatially depending context. ### Claim 14 (depends on 1) The apparatus according to claim 1, wherein the spectral decomposer is configured to set a size of the neighborhood depending on a second signalization in the data stream. ### Claim 15 (depends on 14) The apparatus according to claim 14, wherein the apparatus is configured to decode the second signalization from the data stream at sub-picture granularity. ### Claim 16 (depends on 14) The apparatus according to claim 14, wherein the apparatus is configured to decode the second signalization from the data stream using spatial prediction and/or using entropy decoding using a spatially depending context. ### Claim 17 (depends on 1) The apparatus according to claim 1, wherein the noise reducer is configured to set a strength of the noise reduction depending on a third signalization in the data stream. ### Claim 18 (depends on 17) The apparatus according to claim 17, wherein the apparatus is configured to decode the third signalization from the data stream at sub-picture granularity. ### Claim 19 (depends on 17) The apparatus according to claim 17, wherein the apparatus is configured to decode the third signalization from the data stream using spatial prediction and/or using entropy decoding using a spatially depending context. ### Claim 20 (depends on 17) The apparatus according to claim 17, wherein the apparatus is configured to, in setting a strength of the noise reduction, select a set of possible strengths on the basis of a quantization parameter applied by the reconstructor for reconstructing the predetermined block from the data stream and perform the setting so that the strength is set to one of the set of possible strengths by using the third signalization as an index into the selected set of possible strengths. ### Claim 21 An apparatus for block-based predictive encoding of a picture, comprising: a prediction provider configured to predict a predetermined block of the picture to acquire a first version of a predicted filling of the predetermined block; a spectral decomposer configured to spectrally decompose a region composed of the first version of the predicted filling of the predetermined block and a previously encoded version of a neighborhood of the predetermined block so as to acquire a first spectrum of the region; a noise reducer configured to perform noise reduction on the first spectrum to acquire a second spectrum; a spectral composer configured to subject the second spectrum to spectral composition so as to acquire a modified version of the region including a second version of the predicted filling of the predetermined block; an encoding stage configured to encode the predetermined block into a data stream on the basis of the second version of the predicted filling, wherein the encoding stage is configured to encode a first residual signal into the data stream which corrects a prediction signal for the neighborhood to obtain the previously encoded version of the neighborhood of the predetermined block, and encode a second residual signal into the data stream which corrects the second version of the predicted filling of the predetermined block to obtain an encoded version of the predetermined block. ### Claim 22 (depends on 21) The apparatus according to claim 21, wherein the prediction provider is configured to predict the predetermined block using spatial and/or temporal prediction. ### Claim 23 (depends on 21) The apparatus according to claim 21, wherein the spectral decomposer is configured such that the neighborhood of the predetermined block is a spatial extension of the predetermined block beyond a contiguous half of a circumference of the predetermined block. ### Claim 24 (depends on 21) The apparatus according to claim 21, wherein the predetermined block is rectangular and the circumference of the predetermined block comprises a first, second, third and fourth side sequentially arranged along the circumference of the predetermined block clock-wise, wherein the contiguous half comprises the first and fourth sides. ### Claim 25 (depends on 24) The apparatus according to claim 24, wherein the region is rectangular and the circumference of the region comprises the second and third sides of the predetermined block. ### Claim 26 (depends on 21) The apparatus according to claim 21, wherein the spectral decomposer and the spectral composer are configured such that the spectral decomposition and the spectral composition are mutually inverse with the spectral decomposition being a DST, DCT or wavelet transform. ### Claim 27 (depends on 21) The apparatus according to claim 21, wherein the noise reducer is configured to subject the first spectrum to thresholding. ### Claim 28 (depends on 21) The apparatus according to claim 21, wherein the encoding stage is configured to encode a residual signal into a data stream, the residual signal correcting a prediction error of the second version of the predicted filling of the predetermined block. ### Claim 29 (depends on 28) The apparatus according to claim 28, wherein the encoding stage is configured to determine the prediction error in spatial domain, transform the prediction error into spectral domain and encode the residual signal into the data stream in the spectral domain. ### Claim 30 (depends on 29) The apparatus according to claim 29, wherein the encoding stage is configured to perform the transforming of the residual signal using a transform type also used by the spectral decomposer for the spectral decomposition. ### Claim 31 (depends on 21) The apparatus according to claim 21, wherein the encoding stage is configured to select between using the first version of the predicted filling and the second version of the predicted filling and signal the selection by way of a first signalization in the data stream. ### Claim 32 (depends on 31) The apparatus according to claim 31, wherein the apparatus is configured to perform the selection and encode the first signalization into the data stream at sub-picture granularity. ### Claim 33 (depends on 32) The apparatus according to claim 32, wherein the apparatus is configured to encode the first signalization into the data stream using spatial prediction and/or using entropy encoding using a spatial context. ### Claim 34 (depends on 21) The apparatus according to claim 21, configured to set a size of neighborhood and signalize the size by way of a second signalization in a data stream. ### Claim 35 (depends on 34) The apparatus according to claim 34, wherein the apparatus is configured to perform the setting and encode the second signalization into the data stream at sub-picture granularity. ### Claim 36 (depends on 34) The apparatus according to claim 34, wherein the apparatus is configured to encode the second signalization into the data stream using spatial prediction and/or using entropy encoding using a spatial context. ### Claim 37 (depends on 21) The apparatus according to claim 21, configured to set a strength of the noise reduction and signalize the strength by way of a third signalization in the data stream. ### Claim 38 (depends on 37) The apparatus according to claim 37, wherein the apparatus is configured to perform the setting of the strength and encode the third signalization into the data stream at sub-picture granularity. ### Claim 39 (depends on 37) The apparatus according to claim 37, wherein the apparatus is configured to encode the third signalization into the data stream using spatial prediction and/or using entropy encoding using a spatial context. ### Claim 40 (depends on 37) The apparatus according to claim 37, wherein the apparatus is configured to, in setting a strength of the noise reduction, select a set of possible strengths on the basis of a quantization parameter applied by encoding stage for encoding the predetermined block into the data stream and perform the setting so that the strength is set to one of the set of possible strengths and the third signalization indicates an index into the selected set of possible strengths. ### Claim 41 A method for block-based predictive decoding of a picture, comprising: predicting a predetermined block of the picture to acquire a first version of a predicted filling of the predetermined block; spectrally decomposing a region composed of the first version of the predicted filling of the predetermined block and a reconstructed version of a neighborhood of the predetermined block so as to acquire a first spectrum of the region; performing noise reduction on the first spectrum to acquire a second spectrum; subjecting the second spectrum to spectral composition so as to acquire a modified version of the region including a second version of the predicted filling of the predetermined block; decoding a reconstructed version of the predetermined block from a data stream on the basis of the second version of the predicted filling, wherein a first residual signal is decoded from the data stream and a prediction signal for the neighborhood is corrected using the first residual signal to obtain the reconstructed version of the neighborhood of the predetermined block, and a second residual signal is decoded from the data stream and the second version of the predicted filling of the predetermined block is corrected using the second residual signal to obtain the reconstructed version of the predetermined block. ### Claim 42 A method for block-based predictive encoding of a picture, comprising: predicting a predetermined block of the picture to acquire a first version of a predicted filling of the predetermined block; spectrally decomposing a region composed of the first version of the predicted filling of the predetermined block and a previously encoded version of a neighborhood of the predetermined block so as to acquire a first spectrum of the region; performing noise reduction on the first spectrum to acquire a second spectrum; subjecting the second spectrum to spectral composition so as to acquire a modified version of the region including a second version of the predicted filling of the predetermined block; encoding the predetermined block into a data stream on the basis of the second version of the predicted filling, wherein a first residual signal is encoded into the data stream which corrects a prediction signal for the neighborhood to obtain the previously encoded version of the neighborhood of the predetermined block, and a second residual signal is encoded into the data stream which corrects the second version of the predicted filling of the predetermined block to obtain an encoded version of the predetermined block. ### Claim 43 A non-transitory digital storage medium having a computer program stored thereon to perform the method for block-based predictive decoding of a picture, the method comprising: predicting a predetermined block of the picture to acquire a first version of a predicted filling of the predetermined block; spectrally decomposing a region composed of the first version of the predicted filling of the predetermined block and a reconstructed version of a neighborhood of the predetermined block so as to acquire a first spectrum of the region; performing noise reduction on the first spectrum to acquire a second spectrum; subjecting the second spectrum to spectral composition so as to acquire a modified version of the region including a second version of the predicted filling of the predetermined block; decoding a reconstructed version of the predetermined block from a data stream on the basis of the second version of the predicted filling, when said computer program is run by a computer, wherein a first residual signal is decoded from the data stream and a prediction signal for the neighborhood is corrected using the first residual signal to obtain the reconstructed version of the neighborhood of the predetermined block, and a second residual signal is decoded from the data stream and the second version of the predicted filling of the predetermined block is corrected using the second residual signal to obtain the reconstructed version of the predetermined block. ### Claim 44 A non-transitory digital storage medium having a computer program stored thereon to perform the method for block-based predictive encoding of a picture, the method comprising: predicting a predetermined block of the picture to acquire a first version of a predicted filling of the predetermined block; spectrally decomposing a region composed of the first version of the predicted filling of the predetermined block and a previously encoded version of a neighborhood of the predetermined block so as to acquire a first spectrum of the region; performing noise reduction on the first spectrum to acquire a second spectrum; subjecting the second spectrum to spectral composition so as to acquire a modified version of the region including a second version of the predicted filling of the predetermined block; encoding the predetermined block into a data stream on the basis of the second version of the predicted filling, when said computer program is run by a computer, wherein a first residual signal is encoded into the data stream which corrects a prediction signal for the neighborhood to obtain the previously encoded version of the neighborhood of the predetermined block, and a second residual signal is encoded into the data stream which corrects the second version of the predicted filling of the predetermined block to obtain an encoded version of the predetermined block. ### Claim 45 (depends on 42) A data stream having a picture encoded thereinto, the data stream being generated by a method according to claim 42. ### Claim 46 (depends on 1) The apparatus according to claim 1, wherein the spectral decomposer is configured to spectrally decompose the region so that the spectrum of the region is one of a discrete cosine transform of the region, a discrete sine transform of the region or a wavelet transform of the region. ### Claim 47 (depends on 1) The apparatus according to claim 1, wherein the spectral decomposer is configured to use a two dimensional transform to spectrally decompose the region into a transform coefficient block which consists of transform coefficients which belong to mutually different spectral components and represents the first spectrum of the region wherein a number of transform coefficients of the transform coefficient block is equal to or larger than a number of samples within the region and larger than a number of samples within the predetermined block. ### Claim 48 (depends on 1) The apparatus according to claim 1, wherein the predetermined block is rectangular, the spectral decomposer is configured such that the neighborhood of the predetermined block is a region adjoining outer sides of the predetermined block, and the region is a rectangular region comprising samples of the predetermined block and samples of the L shaped region, and the spectral decomposer is configured to use a two dimensional transform to spectrally decompose the region into a transform coefficient block which consists of transform coefficients which belong to mutually different spectral components and represents the first spectrum of the region wherein a number of transform coefficients of the transform coefficient block is equal to or larger than a number of samples within the region. ### Claim 49 (depends on 1) The apparatus according to claim 1, wherein the predetermined block is rectangular, the spectral decomposer is configured such that the neighborhood of the predetermined block is an L shaped region adjoining an upper and a left-hand side of the predetermined block, and the region is a rectangular region comprising samples of the predetermined block and samples of the L shaped region, and the spectral decomposer is configured to use a two dimensional transform to spectrally decompose the region into a transform coefficient block which consists of transform coefficients which belong to mutually different spectral components and represents the first spectrum of the region wherein a number of transform coefficients of the transform coefficient block is equal to or larger than a number of samples within the region. ### Claim 50 (depends on 1) The apparatus according to claim 1, wherein the neighborhood of the predetermined block is a neighboring region within the picture which spatially adjoins the predetermined block. ### Claim 51 (depends on 50) The apparatus according to claim 50, wherein the spectral decomposer is configured to spectrally decompose the region so that the spectrum of the region is one of a discrete cosine transform of the region, a discrete sine transform of the region or a wavelet transform of the region. ### Claim 52 (depends on 21) The apparatus according to claim 21, wherein the spectral decomposer is configured to spectrally decompose the region so that the spectrum of the region is one of a discrete cosine transform of the region, a discrete sine transform of the region or a wavelet transform of the region. ### Claim 53 (depends on 21) The apparatus according to claim 21, wherein the spectral decomposer is configured to use a two dimensional transform to spectrally decompose the region into a transform coefficient block which consists of transform coefficients which belong to mutually different spectral components and represents the first spectrum of the region wherein a number of transform coefficients of the transform coefficient block is equal to or larger than a number of samples within the region and larger than a number of samples within the predetermined block. ### Claim 54 (depends on 21) The apparatus according to claim 21, wherein the predetermined block is rectangular, the spectral decomposer is configured such that the neighborhood of the predetermined block is a region adjoining outer sides of the predetermined block, and the region is a rectangular region comprising samples of the predetermined block and samples of the L shaped region, and the spectral decomposer is configured to use a two dimensional transform to spectrally decompose the region into a transform coefficient block which consists of transform coefficients which belong to mutually different spectral components and represents the first spectrum of the region wherein a number of transform coefficients of the transform coefficient block is equal to or larger than a number of samples within the region. ### Claim 55 (depends on 21) The apparatus according to claim 21, wherein the predetermined block is rectangular, the spectral decomposer is configured such that the neighborhood of the predetermined block is an L shaped region adjoining an upper and a left-hand side of the predetermined block, and the region is a rectangular region comprising samples of the predetermined block and samples of the L shaped region, and the spectral decomposer is configured to use a two dimensional transform to spectrally decompose the region into a transform coefficient block which consists of transform coefficients which belong to mutually different spectral components and represents the first spectrum of the region wherein a number of transform coefficients of the transform coefficient block is equal to or larger than a number of samples within the region. ### Claim 56 (depends on 21) The apparatus according to claim 21, wherein the neighborhood of the predetermined block is a neighboring region within the picture which spatially adjoins the predetermined block. ### Claim 57 (depends on 52) The apparatus according to claim 52, wherein the spectral decomposer is configured to spectrally decompose the region so that the spectrum of the region is one of a discrete cosine transform of the region, a discrete sine transform of the region or a wavelet transform of the region.