Loop filtering method based on adaptive self-guided filtering

A loop filtering method based on an adaptive self-guided filtering, comprising the following steps:

• S1:
• performing the adaptive self-guided filtering on a reconstruction image, specifically comprising:
• respectively setting a self-guided filtering parameter table at a coding end and a decoding end to store 32 groups of adaptive filtering parameters {r1, ε1, r2, ε2} in pairs, initializing an image level filtering mark as 1, performing the adaptive self-guided filtering on a reconstruction image X by using each group of the adaptive filtering parameters to obtain two filtering results X1 and X2, firstly calculating, by an encoder, local structure information of a current to-be-filtered reconstruction image by using a Sobel operator during filtering, then convoluting the Sobel operator of 3×3 and the reconstruction image X, and calculating a transverse pixel gradient Gx and a longitudinal pixel gradient Gy respectively, by the following formulas:
• G
• x
• =
• [
• -
• 1
• 0
• +
• 1
• -
• 2
• 0
• +
• 2
• -
• 1
• 0
• +
• 1
• ]
• *
• X
• ⁢
• ;
• G
• y
• =
• [
• -
• 1
• -
• 2
• -
• 1
• 0
• 0
• 0
• +
• 1
• +
• 2
• +
• 1
• ]
• *
• X
• ;
• estimating edge information of a current reconstruction image by combining the transverse pixel gradient Gx and the longitudinal pixel gradient Gy by the following formula:
• G
• =
• G
• x
• 2
• +
• G
• y
• 2
• ;
• for one group of the adaptive filtering parameters {r1, ε1} and {r2, ε2} in the self-guided filtering parameter table, performing an adaptive adjustment on a regularization coefficient ε adaptive by using G representing the edge information by the following formulas:
• s
• =
• G
• i
• +
• γ
• 1
• 
• ω
• 
• ⁢
• ∑
• p
• =
• 1
• ω
• i
• ⁢
• ⁢
• G
• p
• +
• γ
• ;
• a
• k
• =
• σ
• k
• 2
• σ
• k
• 2
• +
• ɛ
• s
• ;
• wherein s is a weight coefficient of the regularization coefficient ε at a pixel i, ωi is a filtering window centering at the pixel i, and γ is a constant with a value of 0.01;
• s is always greater than 1 when the pixel i is located at a strong edge of the reconstruction image X, s is always less than 1 when the pixel i is located at a flat region of the reconstruction image X, and an adaptive self-guided filter after the regularization coefficient ε is adjusted by a parameter s has the following characteristics:
• at the strong edge, the regularization coefficient ε becomes smaller ak is closer to 1 and more textures are retained;
• in the flat region, the regularization coefficient ε becomes larger, ak is closer to 0, and bk=(1−ak)μk, that is, bk is closer to an average pixel value μk of the filtering window, wherein the flat region is smoother;
• ak is a coefficient of a local linear model in the filtering window, bk is an intercept of the local linear model in the filtering window, a to-be-filtered pixel and an output pixel in the filtering window has the local linear model, and an output pixel qi at the pixel i is obtained according to the local linear model by the following formulas:
• q
• i
• =
• a
• k
• ⁢
• I
• i
• +
• b
• k
• ·
• ∀
• i
• ∈
• ω
• k
• ;
• wherein Ii is a guide image;
• S2:
• implementing a region-level subspace mapping iteration on each region, specifically comprising:
• S21:
• segmenting the reconstruction image X into a plurality of regions, wherein the plurality of regions have identical size and do not overlap each other, aligning boundaries of the plurality of regions with a boundary of a coding tree unit (CTU), and initializing a region-level filtering mark as 1;
• S22:
• initializing a CTU-level filtering mark of the CTU in the each region as 1, and calculating a mapping coefficient of the each region according to formula (1):
• {α,β}T=(ATA)−1ATb (1);
• wherein A={X1−X, X2−X}, b=Y−X, image matrixes in formula (1) are column vectors, and Y is an original image;
• S23:
• substituting the mapping coefficient {α, β} obtained in S22 into formula (2)
• X
• r
• =
• X
• +
• α
• ⁡
• (
• X
• 1
• -
• X
• )
• +
• β
• ⁡
• (
• X
• 2
• -
• X
• )
• ;
• (
• 2
• )
• to obtain a mapping result Xr, only performing a subspace mapping operation on each CTU with the CTU-level filtering mark of 1 in the each region, determining whether to use the adaptive self-guided filter for the each CTU in a current region according to an RDO, and setting the CTU-level filtering mark of the each CTU as 0 when the CTU-level filtering mark of the each CTU is determined not to use the adaptive self-guided filter;
• S24:
• judging whether there is the CTU-level filtering mark of the each CTU set from 1 to 0 in the each region, when yes, re-calculating a region level mapping coefficient {α, β} for the each CTU of the region level mapping coefficient {α, β} a remaining CTU-level filtering mark is 1 in the each region, returning to the step S23 until the CTU-level filtering mark of the each CTU in the each region is no longer changed or a number of iterations reaches 5 and {α, β} is determined, otherwise, entering a step S25;
• S25:
• mapping the each CTU of the region level mapping coefficient {α, β} the CTU-level filtering mark is 1 in the each region according to the region level mapping coefficient {α, β} finally obtained after an iterative optimization, determining whether to perform a filtering operation on the current region by comparing rate distortion costs before and after a use of the adaptive self-guided filter and region level subspace mapping iterative optimization, setting the region level filtering mark as 1 when the current region is filtered, otherwise setting the region level filtering mark as 0;
• and a calculation formula of a rate distortion cost is RDCost=D+λR, wherein D represents a distortion of a current coding unit, R represents a number of bits consumed by information required by implementing the filtering operation on the current coding unit by coding, and λ is a Lagrange multiplier and is determined by the encoder;
• S3:
• filtering additional information entropy coding:
• S31:
• if an optimal rate distortion cost RDCost1 obtained by traversing 32 groups of the adaptive filtering parameters in the self-guided filter parameter table to perform the adaptive self-guide filtering and the region level subspace mapping optimization is higher than a rate distortion cost RDCost0 without using the adaptive self-guided filter, setting the image level filtering mark of the reconstruction image X as 0, otherwise setting the image level filtering mark as 1;
• S32:
• judging whether the image level filtering mark is 0, when yes, transmitting a code word of 0 to the decoding end to inform the decoding end to implementing the filtering operation on the current reconstruction image;
• otherwise, transmitting a code word of 1 to the decoding end and transmitting information of the adaptive self-guided filter at a same time, wherein the information comprises:
• an optimal filtering parameter index obtained by traversing the self-guided filtering parameter table according to the RDO, the region-level filtering mark of the each region, the CTU-level filtering mark of the each CTU in the each region and the region level mapping coefficient {α, β} of the plurality of regions;
• and
• S4:
• when the decoding end reconstructs a coding image, firstly decoding an image level filtering mark bit of the current reconstruction image, wherein when the image level filtering mark is 0, the current reconstruction image does not need to be filtered and no operation is performed on the reconstruction image X at this time;
• if the image level filtering mark is 1, a self-guided filtering parameter index is continuously decoded, the self-guided filtering parameter corresponding to a decoded index is found from the self-guided filtering parameter table, the self-guided filtering parameter is used to perform the adaptive self-guide filtering to obtain the two filtering results X1 and X2;
• like the coding end, dividing the reconstruction image X into the plurality of regions with a consistent size, continuously decoding the region-level filtering mark bit of the each region and the region level mapping coefficient, wherein the CTU-level filtering mark for the each region of the region level mapping coefficient the region-level filtering mark is 1 is further decoded, a reconstruction pixel before filtering is used as the reconstruction pixel of the each CTU when the CTU-level filtering mark of the each CTU is 0, and when the CTU-level filtering mark of the each CTU is 1, subspace mapping of two groups of adaptive self-guided filtering results of the each CTU is realized by using a corresponding region level mapping coefficient, and the final reconstruction image of the each CTU is replaced by a filtered reconstruction pixel.