# US 11,451,833 B2 · Encoding device, encoding method, decoding device, and decoding method > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,451,833 B2 - Document ID: US11451833B2 - Status: Granted - Kind: B2 - Application number: 16/766,508 - Filed: 2018-11-16 - Published: 2020-12-03 - Granted: 2022-09-20 - Earliest claimed: 2017-12-01 - Assignees: Sony Corporation - Applicants: Sony Corporation - Inventors: Takuro Kawai; Kenichiro Hosokawa; Takafumi Morifuji; Masaru Ikeda; Kenji Kondo; Takahiro Nagano - Web page: https://patentgrep.com/patent/US11451833B2 ## CPC - H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence) - H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/182: the unit being a pixel - H04N 19/184: the unit being bits, e.g. of the compressed video stream - H04N 19/50: using predictive coding (H04N19/61 takes precedence) - H04N 19/82: involving filtering within a prediction loop - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract The present technology relates to an encoding device, an encoding method, a decoding device, and a decoding method that make it possible to accurately restore an image. The encoding device generates a filter image by performing, on a decoded image locally decoded, filter processing of applying a prediction equation including a second-order or higher higher-order term and performing a product-sum calculation of predetermined tap coefficients and pixels of the decoded image. Moreover, the encoding device encodes an original image by using the filter image. The decoding device decodes coded data included in an encoded bit stream by using a filter image, to generate a decoded image. Moreover, the decoding device generates the filter image by performing, on the decoded image, filter processing of applying a prediction equation. The present technology can be applied to a case where encoding and decoding of an image are performed. ## Claims ### Claim 1 A decoding device, comprising: circuitry configured to: receive an encoded bit stream that includes coded data and a plurality of tap coefficients; decode the coded data based on a first filter image, to generate a decoded image; and execute filter processing on the decoded image to generate a second filter image, wherein the filter processing includes: application of a prediction equation including at least one of a second-order term or a higher-order term, wherein the second-order term includes a product of two prediction taps of a plurality of prediction taps, an order of the higher-order term is higher than an order of the second-order term, and the plurality of prediction taps corresponds to a plurality of pixels of the decoded image, and product-sum calculation of the plurality of tap coefficients and the plurality of prediction taps. ### Claim 2 (depends on 1) The decoding device according to claim 1, wherein the prediction equation corresponds to an all combinations prediction equation that includes a plurality of terms, the plurality of terms includes candidate pixels of the plurality of pixels as the plurality of prediction taps, and the circuitry is further configured to: select a set of terms of the plurality of terms of the all combinations prediction equation; and execute the filter processing on a pixel of interest among the plurality of pixels of the decoded image, based on a reduced prediction equation that includes the set of terms selected from the all combinations prediction equation. ### Claim 3 (depends on 2) The decoding device according to claim 2, wherein the circuitry is further configured to execute the filter processing based on the reduced prediction equation including a term of a pixel of the plurality of pixels, the pixel is at a position adjacent to the pixel of interest, and the term is from the plurality of terms of the all combinations prediction equation. ### Claim 4 (depends on 2) The decoding device according to claim 2, wherein the encoded bit stream further includes selection information representing a selection pattern for selection of a term of the set of terms in the reduced prediction equation, and the circuitry is further configured to: parse the selection information included in the encoded bit stream; and execute the filter processing based on the reduced prediction equation of the selection pattern represented by the parsed selection information. ### Claim 5 (depends on 2) The decoding device according to claim 2, wherein the encoded bit stream further includes encoding information regarding encoding of an original image, the circuitry is further configured to: determine, based on the encoding information, a selection pattern from a plurality of selection patterns; select the set of terms based on the selection pattern; and execute the filter processing based on the reduced prediction equation including the set of terms, and the plurality of selection patterns is for selection of a term of the plurality of terms for the reduced prediction equation. ### Claim 6 (depends on 1) The decoding device according to claim 1, wherein the encoded bit stream further includes format information representing an expression format, the expression format expresses with a specific bit depth a tap coefficient of the plurality of tap coefficients of the at least one of the second-order term or the higher-order term, and the circuitry is further configured to: parse the format information included in the encoded bit stream; and, execute the filter processing based on the prediction equation including the plurality of tap coefficients in the expression format represented by the parsed format information. ### Claim 7 (depends on 1) The decoding device according to claim 1, wherein the circuitry is further configured to: parse the plurality of tap coefficients included in the encoded bit stream; and execute the filter processing based on prediction equation including the parsed plurality of tap coefficients. ### Claim 8 (depends on 1) The decoding device according to claim 1, wherein the circuitry is further configured to: execute class classification that classifies a pixel of interest among the plurality of pixels of the decoded image into a class of a plurality of classes; and execute, on the decoded image, the filter processing that includes the application of the prediction equation including the plurality of tap coefficients of the class of the pixel of interest. ### Claim 9 (depends on 1) The decoding device according to claim 1, wherein the circuitry is further configured to decode the coded data with a Coding Unit (CU), and the CU includes one of a Quad-Tree Block Structure or a Quad Tree Plus Binary Tree (QTBT) Block Structure as a unit of processing. ### Claim 10 A decoding method, comprising: receiving an encoded bit stream that includes coded data and a plurality of tap coefficients; decoding the coded data based on a first filter image, to generate a decoded image; and executing filter processing on the decoded image for generating a second filter image, wherein the filter processing includes: applying a prediction equation including at least one of a second-order term or a higher-order term, wherein the second-order term includes a product of two prediction taps of a plurality of prediction taps, an order of the higher-order term is higher than an order of the second-order term, and the plurality of prediction taps corresponds to a plurality of pixels of the decoded image, and executing product-sum calculation of the plurality of tap coefficients and the plurality of prediction taps. ### Claim 11 An encoding device, comprising: circuitry configured to: acquire a locally decoded image; execute filter processing on the locally decoded image to generate a filter image, wherein the filter processing includes: application of a prediction equation including at least one of a second-order term or a higher-order term, wherein the second-order term includes a product of two prediction taps of a plurality of prediction taps, an order of the higher-order term is higher than an order of the second-order term, and the plurality of prediction taps corresponds to a plurality of pixels of the locally decoded image, and product-sum calculation of a plurality of tap coefficients and the plurality of prediction taps; and encode an original image based on the filter image to obtain coded data. ### Claim 12 (depends on 11) The encoding device according to claim 11, wherein the prediction equation corresponds to an all combinations prediction equation that includes a plurality of terms, the plurality of terms includes candidate pixels of the plurality of pixels as the plurality of prediction taps, and the circuitry is further configured to: select a set of terms of the plurality of terms of the all combinations prediction equation; and execute the filter processing on a pixel of interest among the plurality of pixels of the locally decoded image, based on a reduced prediction equation that includes the set of terms selected from the all combinations prediction equation. ### Claim 13 (depends on 12) The encoding device according to claim 12, wherein the circuitry is further configured to execute the filter processing based on the reduced prediction equation including a term of a pixel of the plurality of pixels, the pixel is at a position adjacent to the pixel of interest, and the term is from the plurality of terms of the all combinations prediction equation. ### Claim 14 (depends on 12) The encoding device according to claim 12, wherein the circuitry is further configured to generate an encoded bit stream including: the coded data obtained by the encoding of the original image, and selection information representing a selection pattern for selection of a term of the plurality of terms for the reduced prediction equation. ### Claim 15 (depends on 12) The encoding device according to claim 12, wherein the circuitry is further configured to: determine, based on encoding information regarding the encoding of the original image, a selection pattern from a plurality of selection patterns; select the set of terms based on the selection pattern; and execute the filter processing based on reduced prediction equation including the set of terms, and the plurality of selection patterns is for selection of a term of the plurality of terms for the reduced prediction equation. ### Claim 16 (depends on 11) The encoding device according to claim 11, wherein the circuitry is further configured to determine, for the at least one of the second-order term or the higher-order term, an expression format expressing with a specific bit depth a tap coefficient of the plurality of tap coefficients of the at least one of the second-order term or the higher-order term. ### Claim 17 (depends on 16) The encoding device according to claim 16, wherein the circuitry is further configured to determine, as the expression format of the tap coefficient of the at least one of the second-order term or the higher-order term, an expression format having a bit depth, representing after a decimal point, greater than that of an expression format of a first-order term tap coefficient of the prediction equation. ### Claim 18 (depends on 16) The encoding device according to claim 16, wherein the circuitry is further configured to generate an encoded bit stream including: the coded data obtained by the encoding of the original image, and format information representing the expression format. ### Claim 19 (depends on 11) The encoding device according to claim 11, wherein the circuitry is further configured to generate an encoded bit stream including the coded data and the plurality of tap coefficients. ### Claim 20 (depends on 11) The encoding device according to claim 11, wherein the circuitry is further configured to: obtain the plurality of tap coefficients of each of a plurality of classes based on the locally decoded image and a specific image for the locally decoded image; execute, for each class of the plurality of classes, learning to statistically minimize a prediction error of a predicted value of the specific image, wherein the predicted value of the specific image is based on the application of the prediction equation to the locally decoded image; execute class classification that classifies a pixel of interest among the plurality of pixels of the locally decoded image into a class of the plurality of classes; and execute, on the locally decoded image, the filter processing that includes the application of the prediction equation including the plurality of tap coefficients of the class of the pixel of interest. ### Claim 21 (depends on 11) The encoding device according to claim 11, wherein the circuitry is further configured to encode the original image with a Coding Unit (CU), and the CU includes one of a Quad-Tree Block Structure or a Quad Tree Plus Binary Tree (QTBT) Block Structure as a unit of processing. ### Claim 22 An encoding method, comprising: obtaining a locally decoded image; executing filter processing on the locally decoded image for generating a filter image, wherein the filter processing includes: applying a prediction equation including at least one of a second-order term or a higher-order term, wherein the second-order term includes a product of two prediction taps of a plurality of prediction taps, an order of the higher-order term is higher than an order of the second-order term, and the plurality of prediction taps corresponds to a plurality of pixels of the locally decoded image, and executing product-sum calculation of a plurality of tap coefficients and the plurality of prediction taps; and encoding an original image based on by using the filter image.