Optimal separable adaptive loop filter

An apparatus for encoding and decoding video signals, comprising:

• (a) an encoder comprising a first computer and programming executable on said first computer for encoding video signals;
• and
• (b) a decoder comprising a second computer and programming executable on said second computer for processing video signals encoded by said encoder by performing steps comprising:
• (i) performing entropy decoding;
• (ii) reconstructing macroblocks using inverse quantization, inverse transforms, and prediction to reconstruct each macroblock;
• (iii) executing a deblocking filter;
• (iv) executing a quadtree based adaptive loop filter having separable filters for P and B pictures;
• (v) wherein said separable filters comprise a two dimensional (2D) separable filter as a product of a one-dimensional (1D) vertical filter and a one dimensional (1D) horizontal filter;
• (vi) wherein said 1D horizontal filter and said 1D vertical filter are initially determined from a non-separable 2D filter, followed by iteratively minimizing error for pixels within ON regions of a quadtree structure of said quadtree based adaptive loop filter;
• (vii) wherein said minimizing error is performed by determining a matrix and a cross correlation that assumes said 2D separable filter is not zero phase and performing a transposition on the 2D separable filter and cross correlation;
• and
• (viii) outputting a decoded video signal.

A method of decoding encoded video signals received from a video encoder, the video encoder configured for performing iterations of a quadtree based adaptive loop filter to minimize a sum of squares error, the method comprising:

• performing entropy decoding of the video signals;
• reconstructing macroblocks using inverse quantization, inverse transforms, and prediction to reconstruct each macroblock;
• executing a deblocking filter;
• executing, on a computer, a quadtree based adaptive loop filter having separable filters for P and B pictures, and one non-separable filter for I pictures;
• wherein said separable filters are a two dimensional (2D) separable filter determined as a product of a one-dimensional (1D) vertical filter and a one dimensional (1D) horizontal filter,
• wherein said 1D horizontal filter and said 1D vertical filter are initially determined from a non-separable 2D filter, followed by iteratively minimizing error for pixels within ON regions of a quadtree structure of said quadtree based adaptive loop filter;
• wherein said minimizing error is performed by determining a matrix and a cross correlation that assumes said separable 2D filter is not zero phase and performing a transposition on the 2D separable filter and cross correlation;
• and
• outputting a decoded video signal.