Back to results
US 6,212,235 B1Granted
Video encoder and decoder using motion-based segmentation and merging
AssigneeNokia Mobile Phones Ltd.
InventorsJacek Nieweglowski, Marta Karczewicz
Life of this patent
Estimating the term…
Abstract
This invention relates to motion compensated (MC) coding of video and to a MC prediction scheme which allows fast and compact encoding of motion vector fields retaining at the same time very low prediction error. By reducing prediction error and number of bits needed for representation of motion vector field, substantial savings of bit rate are achieved. Reduction of bit rate needed to represent motion field is achieved by merging segments in video frames, by adaptation of motion field model and by utilization of motion field model based on orthogonal polynomials.
Claims33
Encoder for performing motion compensated encoding of video data, comprising:
- motion field estimating means, having an input for receiving a first video data frame In and a reference frame Rref, said motion field estimating means being arranged to estimate a motion vector field describing scene motion displacements of video frame pixels;
- motion field encoding means having an input to receive from said motion field estimating means said estimated motion vector field;
- partitioning information indicating partitioning of a video frame into at least two segments said segments being a first segment Si and a second segment Sj;
- said motion field encoding means being arranged to obtain compressed motion information comprising first motion coefficients representing said motion vector field;
- motion compensated prediction means for predicting a predicted video data frame based on said reference frame Rref and said compressed motion information;
- computing means having an input for receiving said first video data frame and said predicted video data frame, said computing means being arranged to calculate a prediction error frame based on said predicted video data frame and on said first video data frame;
- prediction error encoding means for encoding said prediction error frame;
- wherein said motion encoding means further comprises:
- means for calculating for each segment a distortion matrix E and a distortion vector y such that a predefined measure ΔE for distortion in each segment, due to approximating said motion vector field as coefficients ci of a set of polynomial basis function fi, is a function of (Ec−y), c being a vector of said motion coefficients ci;
- means for decomposing said distortion matrix E into a first matrix Q and a second matrix R such that
- det Q≠0 and
- Q R=E,
- a subset of the set of all columns of matrix Q being a basis of a vector space defined by all possible linear combinations of all column vectors of matrix E, columns of matrix Q being orthogonal to each other;
- means for calculating an auxiliary vector z according to z=Q−1y, Q−1 being the inverse matrix of said first matrix Q;
- means for generating for each segment a column extended matrix A comprising the columns of matrix R and vector z as an additional column, and for selecting all rows of matrix A which have elements unequal to zero in all columns due to matrix R;
- means for merging segments based on selective combination of segments producing an increase in said prediction error within a certain limit;
- means for generating a row extended matrix B comprising said selected rows of matrix A of said first segment Si and said selected rows of matrix A of said second segment Sj;
- means for performing a series of multiplications of rows of matrix B with scalars unequal to zero and additions of rows of matrix B in order to obtain a modified matrix B′ having in the columns due to matrix R as many rows as possible filled with zeros;
- orthogonalising means receiving one of said matrices A, B and B′ as an input matrix M, said orthogonalising means being arranged to replace said polynomial basis functions fi by orthogonal basis functions {tilde over (f)}i and to calculate second motion coefficients {tilde over (c)} using said orthogonal basis functions and said input matrix M;
- and
- quantisation means for quantising said second coefficients {tilde over (c)}.
18 more elements
Encoder for performing motion compensated encoding of video data, comprising:
- motion field estimating means, having an input for receiving a first video data frame In and a reference frame Rref, said motion field estimating means being arranged to estimate a motion vector field describing motion displacements of video frame pixels;
- motion field encoding means having an input to receive from said motion field estimating means said estimated motion vector field;
- partitioning information indicating partitioning of a video frame into at least two segments said segments being a first segment Si and a second segment Sj;
- said motion field encoding means being arranged to obtain compressed motion information comprising first motion coefficients representing said motion vector field;
- motion compensated prediction means for predicting a predicted video data frame based on said reference frame Rref and said compressed motion information;
- computing means having an input for receiving said first video data frame and said predicted video data frame, said computing means being arranged to calculate a prediction error frame based on said predicted video data frame and on said first video data frame;
- prediction error encoding means for encoding said prediction error frame;
- wherein said motion encoding means further comprises:
- means, for calculating for each segment a distortion matrix E and a distortion vector y such that a predefined measure ΔE distortion in each segment, due to approximating said motion vector field as coefficients ci of a set of polynomial basis function fi, is a function of (E c−y), c being a vector of said motion coefficients ci;
- means for decomposing said distortion matrix E into a first matrix Q and a second matrix R such that
- det Q≠0, and
- Q R=E,
- a subset of the set of all columns of matrix Q being a basis of a vector space defined by all possible linear combinations of all column vectors of matrix E, columns of matrix Q being orthogonal to each other;
- means for calculating an auxiliary vector z according to z=Q−1y, Q−1 being the inverse matrix of said first matrix Q;
- means for generating for each segment a column extended matrix A comprising the columns of matrix R and vector z as an additional column, and for selecting all rows of matrix A which have elements unequal to zero in all columns due to matrix R;
- means for merging segments based on selective combination of segments producing an increase in said prediction error within a certain limit;
- means for generating a row extended matrix B comprising said selected rows of matrix A of said first segment Si and said selected rows of matrix A of said second segment Sj;
- means for performing a series of multiplications of rows of matrix B with scalars unequal to zero and additions of rows of matrix B in order to obtain a modified matrix B′ having in the columns due to matrix R as many rows as possible filled with zeros;
- orthogonalising means receiving on of said matrices A, B and B′ as an input matrix M, said orthogonalising means being arranged to replace said polynomial basis functions fi by orthogonal basis functions {tilde over (f)}i and to modify said input matrix to a third matrix {tilde over (M)} corresponding to said orthogonal basis functions;
- removing means having an input for receiving of said third matrix {tilde over (M)}, said removing means being arranged to modify said third matrix by removing from said third matrix the ith column due to R corresponding to the ith basis function of said orthogonal basis functions, and said removing means having an output to provide a matrix {circumflex over (M)};
- means for computing second motion coefficients {tilde over (c)} using said fourth matrix {circumflex over (M)};
- and
- quantisation means for quantizing said second motion coefficients {tilde over (c)}.
20 more elements
Method of motion compensated encoding of video data, comprising the steps:
- a) receiving a first video data In and a reference frame Rref, estimating a motion vector field describing motion displacements of video frame pixels, and outputting said first video frame, said motion vector field and said reference frame Rref;
- b) receiving said estimated motion vector field;
- partitioning information indicating partitioning of a video frame into at least two segments, said segments being a first segment Si and a second segment Sj, for obtaining compressed motion information comprising first motion coefficients representing said motion vector field;
- c) predicting a predicted video data frame based on said reference frame Rref and said compressed motion information;
- d) receiving said first video data frame to calculate a prediction error frame based on said predicted video data frame and on said first video data frame;
- e) encoding said prediction error frame;
- wherein said method further comprises the steps:
- g) calculating for each segment a distortion matrix E and a distortion vector y such that a predefined measure ΔE for distortion in each segment, due to approximating said motion vector field as coefficients ci of a set of polynomial basis functions fi, is a function of (E c−y), c being a vector of said motion coefficients ci;
- h) decomposing said distortion matrix E into a first matrix Q and a second matrix R such that
- det Q≠0 and
- Q R=E,
- a subset of the set of all columns of matrix Q being a basis of a vector space defined by all possible linear combinations of all column vectors of matrix E, columns of matrix Q being orthogonal to each other;
- i) calculating an auxiliary vector z according to z=Q−1 y, Q−1 being the inverse matrix of said first matrix Q;
- j) generating for each segment a column extended matrix A comprising the columns of matrix R and vector z as an additional column, and for selecting all rows of matrix A which have elements unequal to zero in all columns due to R;
- k) merging segments based on selective combination of segments producing an increase in said prediction error within a certain limit;
- l) generating a row extended matrix B comprising said selected rows of matrix A of said first segment Si and said selected rows of matrix A of said second segment Sj;
- m) performing a series of multiplications of rows of matrix B with scalars unequal to zero and additions of rows of matrix B in order to obtain a modified matrix B′ having in the columns due to matrix R as many rows as possible filled with zeros;
- o) receiving one of said matrices A, B and B′ as an input matrix M in order to replace said polynomial basis functions fi by orthogonal basis functions {tilde over (f)}i and to calculate second motion coefficients {tilde over (c)} using said orthogonal basis functions and said input matrix M;
- and
- q) quantising said second motion coefficients {tilde over (c)}.
17 more elements
Method of motion compensated encoding of video data, comprising the steps:
- a) receiving a first video data frame In and a reference frame Rref, estimating a motion vector field describing motion displacements of video frame pixels, and outputting said first video frame, said motion vector field and said reference frame Rref;
- b) receiving said estimated motion vector field;
- partitioning information indicating partitioning of a video frame into at least two segments, said segments being a first segment Si and a second segment Sj, for obtaining compressed motion information comprising first motion coefficients representing said motion vector field;
- c) predicting a predicted video data frame based on said reference frame Rref and said compressed motion information;
- d) receiving said first video data frame to calculate a prediction error frame based on said predicted video data frame and on said first video data frame;
- e) encoding said prediction error frame;
- wherein said method further comprises the steps:
- g) calculating for each segment a distortion matrix E and a distortion vector y such that a predefined measure ΔE for distortion in each segment, due to approximating said motion vector field as coefficients ci of a set of polynomial basis functions fi, is a function of (E c−y), c being a vector of said motion coefficients ci;
- h) decomposing said distortion matrix E into a first matrix Q and a second matrix R such that
- det Q≠0 and
- Q R=E,
- a subset of the set of all columns of matrix Q being a basis of a vector space defined by all possible linear combinations of all column vectors of matrix E, columns of matrix Q being orthogonal to each other;
- i) calculating an auxiliary vector z according to z=Q−1 y, Q−1 being the inverse matrix of said first matrix Q;
- j) generating for each segment a column extended matrix A comprising the columns of matrix R and vector z as an additional column, and for selecting all rows of matrix A which have elements unequal to zero in all columns due to matrix R;
- k) merging segments based on selective combination of segments producing an increase in said prediction error within a certain limit;
- l) generating a row extended matrix B comprising said selected rows of matrix A of said first segment Si and said selected rows of matrix A of said second segment Sj;
- m) performing a series of multiplications of rows of matrix B with scalars unequal to zero and additions of rows of matrix B in order to obtain a modified matrix B′ having in the columns due to matrix R as many rows as possible filled with zeros;
- n) receiving one of said matrices A, B, and B′ as an input matrix M in order to replace said polynomial basis functions fi by orthogonal basis functions {tilde over (f)}i and to modify said input matrix to a third matrix {tilde over (M)} corresponding to said orthogonal basis functions;
- o) receiving said third matrix {tilde over (M)} to modify said third matrix {tilde over (M)} to a fourth matrix {circumflex over (M)} by removing from said third matrix {tilde over (M)} the ith column due to matrix R corresponding to the ith basis function of said orthogonal basis functions, and outputting said fourth matrix {circumflex over (M)};
- p) computing second motion coefficients {tilde over (c)} using said fourth matrix {circumflex over (M)};
- and
- q) quantising said second motion coefficients {tilde over (c)}.
19 more elements
Decoder for decoding of motion compensation encoded video data, said decoder comprising:
- means for storing a video data frame;
- means for predicting a video data frame based on said stored video data frame and on received motion information;
- means for decoding received prediction error data and obtaining a prediction error frame;
- and
- means for calculating and outputting an updated video data frame based on said predicted video data frame and said decoded prediction error frame, and storing the updated video data frame in said storing means;
- said means for predicting a video data frame comprising:
- means for demultiplexing received motion data into at least two of the following:
- data indicating partitioning of said updated video data frame into segments Sk, data indicating a selection of basis functions from a set of motion field model basis functions, and data indicating coefficients of selected basis functions;
- means for reconstructing said motion vector field in each segment Sk from a linear combination of said selected basis functions and coefficients;
- and
- means for calculating said prediction frame based on said reconstructed motion vector field and based on said stored video data frame.
8 more elements
Decoder for decoding of motion compensation encoded video data, said decoder comprising:
- means for storing a video data frame;
- means for predicting a video data frame based on said stored video data frame and on received motion information;
- means for decoding received prediction error data and obtaining a prediction error frame;
- and
- means for calculating and outputting an updated video data frame based on said predicted video data frame and said decoded prediction error frame, and storing the undated video data frame in said storing means;
- said means for predicting a video data frame comprising:
- means for demultiplexing received motion data into at least two of the following:
- data indicating partitioning of said undated video data frame into segments Sk, data indicating a selection of basis functions from a set of motion field model basis functions, and data indicating coefficients of selected basis functions;
- means for reconstructing said motion vector field in each segment Sk from a linear combination of said selected basis functions and coefficients;
- and
- means for calculating said prediction frame based on said reconstructed motion vector field and based on said stored video data frame;
- wherein said means for reconstructing said motion vector field is adapted to calculate said set of motion field model basis functions for each segment Sk as being orthogonal to each other with respect to an area determined by the shape of the segment Sk.
9 more elements
System for handling video data, comprising an encoder for performing motion compensated encoding of video data and a decoder for decoding motion compensated video data, said encoder comprising:
- motion field estimating means, having an input for receiving a first video data frame In and a reference frame Rref, said motion field estimating means being arranged to estimate a motion vector field describing motion displacements of video frame pixels;
- motion field encoding means having an input to receive from said motion field estimating means said estimated motion vector field;
- partitioning information indicating partitioning of a video frame into at least two segments said segments being a first segment Si and a second segment Sj;
- said motion field encoding means being arranged to obtain compressed motion information comprising first motion coefficients representing said motion vector field;
- motion compensated prediction means for predicting a predicted video data frame based on said reference frame Rref and said compressed motion information;
- computing means having an input for receiving said first video data frame and said predicted video data frame, said computing means being arranged to calculate a prediction error frame based on said predicted video data frame and on said first video data frame;
- prediction error encoding means for encoding said prediction error frame;
- said motion encoding means further comprising:
- means for calculating for each segment a distortion matrix E and a distortion vector y such that a predefined measure ΔE for distortion in each segment, due to approximating said motion vector field as coefficients ci of a set of polynomial basis functions fi, is a function of (E c−y), c being a vector of said motion coefficients ci;
- means for decomposing said distortion matrix E into a first matrix Q and a second matrix R such that
- det Q≠0 and
- Q R=E,
- a subset of the set of all columns of matrix Q being a basis of a vector space defined by all possible linear combinations of all column vectors of matrix E, columns of matrix Q being orthogonal to each other;
- means for calculating an auxiliary vector z according to z=Q−1 y, Q−1 being the inverse matrix of said first matrix Q;
- means for generating for each segment a column extended matrix A comprising the columns of matrix R and vector z as an additional column, and for selecting all rows of matrix A which have elements unequal to zero in all columns due to R;
- means for merging segments based on selective combination of segments producing an increase in said prediction error within a certain limit;
- means for generating a row extended matrix B comprising said selected rows of matrix A of said first segment Si and said selected rows of matrix A of said second segment Sj;
- means for performing a series of multiplications of rows of matrix B with scalars unequal to zero and additions of rows of matrix B in order to obtain a modified matrix B′ having in the columns due to matrix R as many rows as possible filled with zeros;
- orthogonalising means receiving one of said matrices A, B and B′ as an input matrix M, said orthogonalising means being arranged to replace said polynomial basis function fi by orthogonal basis functions {tilde over (f)}i and to calculate second motion coefficients {tilde over (c)} using said orthogonal basis functions and said input matrix M;
- and
- quantisation means for quantizing said second coefficients {tilde over (c)};
- and said decoder comprising:
- means for storing a video data frame;
- means for predicting video data fame based on said stored video data frame and on received motion information;
- means for decoding received prediction error data and obtaining a prediction error frame;
- and
- means for calculating and outputting an updated video data frame based on said predicted video data frame and said decoded prediction error frame, and storing the updated video data frame in said storing means;
- said means for predicting a video data frame further comprising:
- means for demultiplexing received motion data into at least two of the following:
- data indicating partitioning of said updated video data frame into segments Sk, data indicating a selection of basis functions from a set of motion field model basis functions, and data indicating coefficients of selected basis functions;
- means for reconstructing said motion vector field in each segment Sk from a linear combination of said selected basis functions and corresponding coefficients;
- and
- means for calculating said prediction frame based on said reconstructed motion vector field and based on said stored video data frame.
30 more elements
System for handling video data, comprising an encoder for performing motion compensated encoding of video data and a decoder for decoding motion compensated video data, said encoder comprising:
- motion field estimating means having an input for receiving a first video data frame In and a reference frame Rref, said motion field estimating means being arranged to estimate a motion vector field describing motion displacements of video frame pixels;
- motion field encoding means having an input to receive from said motion field estimating means said estimated motion vector field;
- partitioning information indicating partitioning of a video frame into at least two segments said segments being a first segment Si and a second segment Sj;
- said motion field encoding means being arranged to obtain compressed motion information comprising first motion coefficients representing said motion vector field;
- motion compensated prediction means for predicting a predicted video data frame based on said reference frame Rref and said compressed motion information;
- computing means having an input for receiving said first video data frame and said predicted video data frame, said computing means being arranged to calculate a prediction error frame based on said predicted video data frame and on said first video data frame;
- prediction error encoding means for encoding said prediction error frame;
- said motion encoding means further comprising:
- means for calculating for each segment a distortion matrix E and a distortion vector y such that a predefined measure ΔE for distortion in each segment, due to approximating said motion vector field as coefficients ci of a set of polynomial basis functions fi, is a function of (E c−y), c being a vector of said motion coefficients ci;
- means for decomposing said distortion matrix E into a first matrix Q and a second matrix R such that
- det Q≠0, and
- Q R=E,
- a subset of the set of all columns of matrix Q being a basis of a vector space defined by all possible linear combinations of all column vectors of matrix E, columns of matrix Q being orthogonal to each other;
- means for calculating an auxiliary vector z according to z=Q−1 y, Q−1 being the inverse matrix of said first matrix Q;
- means for generating for each segment a column extended matrix A comprising the columns of matrix R and vector z as an additional column, and for selecting all rows of matrix A which have elements unequal to zero in all columns due to matrix R;
- means for merging segments based on selective combination of segments producing an increase in said prediction error within a certain limit;
- means for generating a row extended matrix B comprising said selected rows of matrix A of said first segment Si said selected rows of matrix A of said second segment Sj;
- means for performing a series of multiplications of rows of matrix B with scalars unequal to zero and additions of rows of matrix B in order to obtain a modified matrix B′ having in the columns due to matrix R as many rows as possible filled with zeros;
- orthogonalising means receiving one of said matrices A, B and B′ as an input matrix M, said orthogonalising means being arranged to replace said polynomial basis functions fi by orthogonal basis functions {tilde over (f)}i and to modify said input matrix to a third matrix {tilde over (M)} corresponding to said orthogonal basis functions;
- removing means having an input for receiving said third matrix {tilde over (M)}, said removing means being arranged to modify said third matrix by removing from said third matrix the ith column due to R corresponding to the ith basis function of said orthogonal basis functions, and said removing means having an output to provide a fourth matrix {circumflex over (M)};
- means for computing second motion coefficients {tilde over (c)} using said fourth matrix {circumflex over (M)};
- and
- quantisation means for quantizing said second motion coefficients {tilde over (c)};
- and said decoder comprising:
- means for storing a video data frame;
- means for predicting a video data frame based on said stored video data frame and on received motion information;
- means for decoding received prediction error data and obtaining a prediction error frame;
- and
- means for calculating and outputting an updated video data frame based on said predicted video data frame and said decoded prediction error frame, and storing the updated video data frame in said storing means;
- said means for predicting a video data frame comprising:
- means for demultiplexing received motion data into at least two of the following:
- data concerning a partitioning of said updated video data frame into segments Sk, data concerning a selection of basis functions from a set of motion field model basis functions, and data concerning coefficients of selected basis functions;
- means for reconstructing said motion vector field in each segment Sk from a linear combination of said selected basis functions and corresponding coefficients;
- and
- means for calculating said prediction frame based on said reconstructed motion vector field and based on said stored video data frame.
32 more elements
Method of decoding motion compensation encoded video data, said method comprising:
- storing a video data frame;
- predicting a video data frame based on said stored video data frame and on received motion information;
- decoding received prediction error data and obtaining a prediction error frame;
- and
- calculating and outputting an updated video data frame based on said predicted video data frame and said decoded prediction error frame, and storing the updated video data frame;
- predicting of a video data frame comprising:
- demultiplexing received motion data into at least two of the following:
- basis function selection data indicating a selection of basis functions from a set of motion field model basis functions and coefficient values for selected basis functions, and partitioning data indicating the partitioning of said updated video data frame into segments Sk;
- reconstructing said motion vector field in each segment Sk from a linear combination of said selected basis functions and corresponding coefficient values;
- and
- calculating said prediction frame based on said reconstructed motion vector field and based on said stored video data frame.
8 more elements
Decoder for decoding of motion compensation encoded video data, said decoder comprising;
- means for storing a video data frame;
- means for predicting a video data frame based on said stored video data frame and on received motion information;
- means for decoding received prediction error data and obtaining a prediction error frame;
- and
- means for calculating and outputting an updated video data frame based on said predicted video data frame and said decoded prediction error frame, and storing the updated video data frame in said storing means;
- said means for predicting a video data frame comprising:
- means for demultiplexing received motion data into at least basis function selection data indicating a selection of basis functions from a set of motion field model basis functions and coefficient values for selected basis functions, and partitioning data indicating the partitioning of said updated video data frame into segments Sk;
- means for reconstructing said motion vector field in each segment Sk from a linear combination of said selected basis functions and corresponding coefficient values;
- and
- means for calculating said prediction frame based on said reconstructed motion vector field and based on said stored video data frame.
7 more elements
Dependencies are inferred from the claim text.