# US 8,542,736 B2 · Method and apparatus for video coding using prediction data refinement > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,542,736 B2 - Document ID: US8542736B2 - Status: Granted - Kind: B2 - Application number: 12/311,036 - Filed: 2007-10-11 - Published: 2009-09-24 - Granted: 2013-09-24 - Earliest claimed: 2006-10-18 - Assignees: Thomson Licensing - Applicants: Shay Har-Noy; Oscar Divorra Escoda; Peng Yin; Cristina Gomila - Inventors: Shay Har-Noy; Oscar Divorra Escoda; Peng Yin; Cristina Gomila - Web page: https://patentgrep.com/patent/US8542736B2 ## CPC - H04N 19/82: involving filtering within a prediction loop - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/11: among a plurality of spatial predictive coding modes - H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) - H04N 19/137: Motion inside a coding unit, e.g. average field, frame or block difference - H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) - H04N 19/593: involving spatial prediction techniques - H04N 19/61: in combination with predictive coding - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract There are provided methods and apparatus for video coding using prediction data refinement. An apparatus includes an encoder for encoding an image region of a picture. The encoder has a prediction refinement filter for refining at least one of an intra prediction and an inter prediction for the image region. The prediction refinement filter refines the inter prediction for the image region using at least one of previously decoded data and previously encoded data, the previously decoded data and the previously encoded data corresponding to pixel values in neighboring regions with respect to the image region. ## Claims ### Claim 1 An apparatus, comprising: an encoder for encoding an image region of a picture, said encoder having a prediction refinement filter, operating in a coding loop on a prediction for an input image region and before a residual error is generated, for refining any of an intra prediction and an inter prediction for the image region, wherein said prediction refinement filter refines the inter prediction for the image region using at least one of previously decoded data and previously encoded data, the previously decoded data and the previously encoded data corresponding to pixel values in neighboring regions with respect to the image region, wherein said encoder further includes an inverse transformer and inverse quantizer unit and a combiner, the inverse transformer and inverse quantizer unit having an output directly connected to a first input of the combiner, the combiner having a second input directly connected to an output of the prediction refinement filter, wherein the encoder further comprises a deblocking filter, and an output of the combiner is directly connected to both an input of the prediction refinement filter and an input of the deblocking filter. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the image region corresponds to any of multi-view video content for a same or similar scene, single-view video content, and a scalable layer from a set of scalable layers for the same scene. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein said prediction refinement filter is adaptively enabled or disabled depending upon at least one of data characteristics and data statistics corresponding to at least one of the image region and neighboring regions. ### Claim 4 (depends on 3) The apparatus of claim 3, wherein the at least one of data characteristics and data statistics comprise at least one of coding modes, motion data or residue data. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein said prediction refinement filter is selectively applied in a second pass encoding of the picture without being applied in a first pass encoding of the picture. ### Claim 6 A method, comprising: encoding an image region of a picture using a prediction refinement filter, operating in a coding loop on a prediction for an input image region and before a residual error is generated, to refine any of an intra prediction and an inter prediction for the image region, wherein said prediction refinement filter refines the inter prediction for the image region using at least one of previously decoded data and previously encoded data, the previously decoded data and the previously encoded data corresponding to pixel values in neighboring regions with respect to the image region, wherein said encoding step is performed in an encoder that includes the prediction refinement filter, the encoder further including an inverse transformer and inverse quantizer unit and a combiner, the inverse transformer and inverse quantizer unit having an output directly connected to a first input of the combiner, the combiner having a second input directly connected to an output of the prediction refinement filter, wherein the encoder further comprises a deblocking filter, and an output of the combiner is directly connected to both an input of the prediction refinement filter and an input of the deblocking filter. ### Claim 7 (depends on 6) The method of claim 6, wherein the image block corresponds to any of multi-view video content for a same or similar scene, single-view video content, and a scalable layer from a set of scalable layers for the same scene. ### Claim 8 (depends on 6) The method of claim 6, wherein the prediction refinement filter is adaptively enabled or disabled depending upon at least one of data characteristics and data statistics corresponding to the image block. ### Claim 9 (depends on 8) The method of claim 8, wherein the at least one of data characteristics and data statistics comprise at least one of coding modes, motion data or residue data. ### Claim 10 (depends on 6) The method of claim 6, wherein the prediction refinement filter is selectively applied in a second pass encoding of the picture without being applied in a first pass encoding of the picture. ### Claim 11 An apparatus, comprising: a decoder for decoding an image region of a picture, said decoder having a prediction refinement filter, operating in a decoding loop on a prediction for an input image region and before a residual error is generated, for refining any of an intra prediction and an inter prediction for the image region, wherein said prediction refinement filter refines the inter prediction for the image region using previously decoded data corresponding to pixel values in neighboring regions with respect to the image region, wherein said decoder further includes an inverse transformer and inverse quantizer unit and a combiner, the inverse transformer and inverse quantizer unit having an output directly connected to a first input of the combiner, the combiner having a second input directly connected to an output of the prediction refinement filter, wherein the decoder further comprises a deblocking filter, and an output of the combiner is directly connected to both an input of the prediction refinement filter and an input of the deblocking filter. ### Claim 12 (depends on 11) The apparatus of claim 11, wherein said prediction refinement filter is adaptively enabled or disabled depending upon at least one of data characteristics and data statistics corresponding to the image block. ### Claim 13 (depends on 12) The apparatus of claim 12, wherein enablement information or disablement information for said prediction refinement filter is decoded from at least one high level syntax element. ### Claim 14 (depends on 13) The apparatus of claim 13, wherein the at least one high level syntax element is located at at least one of a slice header level, a Supplemental Enhancement Information (SEI) level, a picture parameter set level, a sequence parameter set level and a network abstraction layer unit header level. ### Claim 15 (depends on 11) The apparatus of claim 11, wherein said prediction refinement filter refines the intra prediction for the image region using at least one of previously decoded data and previously encoded data, the previously decoded data and the previously encoded data corresponding to pixel values in neighboring regions with respect to the image region. ### Claim 16 A method, comprising: decoding an image region of a picture using a prediction refinement filter, operating in a decoding loop on a prediction for an input image region and before a residual error is generated, to refine any of an intra prediction and an inter prediction for the image region, wherein said prediction refinement filter refines the inter prediction for the image region using previously decoded data corresponding to pixel values in neighboring regions with respect to the image region, wherein said decoding step is performed in a decoder that includes the prediction refinement filter, the decoder further including an inverse transformer and inverse quantizer unit and a combiner, the inverse transformer and inverse quantizer unit having an output directly connected to a first input of the combiner, the combiner having a second input directly connected to an output of the prediction refinement filter, wherein the decoder further comprises a deblocking filter, and an output of the combiner is directly connected to both an input of the prediction refinement filter and an input of the deblocking filter. ### Claim 17 (depends on 16) The method of claim 16, wherein the prediction refinement filter is applied with respect to an iterative missing data estimation method and a non-iterative missing data estimation method. ### Claim 18 (depends on 16) The method of claim 16, wherein the prediction refinement filter is adaptively enabled or disabled depending upon at least one of data characteristics and data statistics corresponding to the image block. ### Claim 19 (depends on 18) The method of claim 18, wherein the at least one of data characteristics and data statistics comprise at least one of coding modes, motion data or residue data. ### Claim 20 (depends on 16) The method of claim 16, wherein said decoding step further comprises decoding side information to adaptively determine parameters of the prediction refinement filter corresponding to the image block, the side information being decoded from at least one of a macroblock level, a slice level, a picture level, and a sequence level. ### Claim 21 A non-transitory storage media readable by a machine and having executable instructions encoded thereupon to perform an encoding method, the method steps comprising: encoding an image region of a picture using a prediction refinement filter, operating in a coding loop on a prediction for an input image region and before a residual error is generated, to refine any of an intra prediction and an inter prediction for the image region, wherein said prediction refinement filter refines the inter prediction for the image region using at least one of previously decoded data and previously encoded data, the previously decoded data and the previously encoded data corresponding to pixel values in neighboring regions with respect to the image region, wherein said encoding step is performed in an encoder that includes the prediction refinement filter, the encoder further including an inverse transformer and inverse quantizer unit and a combiner, the inverse transformer and inverse quantizer unit having an output directly connected to a first input of the combiner, the combiner having a second input directly connected to an output of the prediction refinement filter, wherein the encoder further comprises a deblocking filter, and an output of the combiner is directly connected to both an input of the prediction refinement filter and an input of the deblocking filter. ### Claim 22 (depends on 21) The non-transitory storage media of claim 21, wherein the image region is at least one of a block, a macroblock, a union of blocks, and a union of macroblocks. ### Claim 23 (depends on 21) The non-transitory storage media of claim 21, wherein the image region corresponds to any of multi-view video content for a same or similar scene, single-view video content, and a scalable layer from a set of scalable layers for the same scene. ### Claim 24 (depends on 23) The non-transitory storage media of claim 23, wherein the at least one of data characteristics and data statistics comprise at least one of coding modes, motion data or residue data. ### Claim 25 (depends on 23) The non-transitory storage media of claim 23, wherein enablement information or disablement information for said prediction refinement filter is determined such that at least one of a distortion measure and a coding cost measure is minimized. ### Claim 26 (depends on 1) The apparatus of claim 1, wherein the prediction refinement filter comprises a first input and a second input, and the apparatus further comprises: an intra prediction module having an output connected to the first input of the prediction refinement filter for providing the intra prediction thereto; and a motion compensator having an output connected to the second input of the prediction refinement filter for providing the inter prediction thereto. ### Claim 27 (depends on 6) The method of claim 6, further comprising providing an encoder to perform said encoding step, said encoder comprising the prediction refinement filter having a first input and a second input, an intra prediction module having an output connected to the first input of the prediction refinement filter for providing the intra prediction thereto, and a motion compensator having an output connected to the second input of the prediction refinement filter for providing the inter prediction thereto. ### Claim 28 (depends on 11) The apparatus of claim 11, wherein the prediction refinement filter comprises a first input and a second input, and the apparatus further comprises: an intra prediction module having an output connected to the first input of the prediction refinement filter for providing the intra prediction thereto; and a motion compensator having an output connected to the second input of the prediction refinement filter for providing the inter prediction thereto. ### Claim 29 (depends on 16) The method of claim 16, further comprising providing a decoder to perform said decoding step, said decoder comprising the prediction refinement filter having a first input and a second input, an intra prediction module having an output connected to the first input of the prediction refinement filter for providing the intra prediction thereto, and a motion compensator having an output connected to the second input of the prediction refinement filter for providing the inter prediction thereto. ### Claim 30 (depends on 21) The non-transitory storage media of claim 21, wherein the image region is encoded by an encoder, the encoder comprising the prediction refinement filter having a first input and a second input, an intra prediction module having an output connected to the first input of the prediction refinement filter for providing the intra prediction thereto, and a motion compensator having an output connected to the second input of the prediction refinement filter for providing the inter prediction thereto.