Back to results
US 2019/0158843 A1Published application
BLOCK-BASED Optical Flow Estimation FOR MOTION COMPENSATED PREDICTION IN VIDEO CODING
AssigneeGoogle Inc.
InventorsYaowu Xu, Bohan Li, Jingning Han
Life of this patent
Estimating the term…
Abstract
An optical flow reference frame portion (e.g., a block or an entire frame) is generated that can be used for inter prediction of blocks of a current frame in a video sequence. A forward reference frame and a backward reference frame are used in an optical flow estimation that produces a respective motion field for pixels of a current frame. The motion fields are used to warp some or all pixels of the reference frames to the pixels of the current frame. The warped reference frame pixels are blended to form the optical flow reference frame portion. The inter prediction may be performed as part of encoding or decoding portions of the current frame.
Claims20
A method, comprising:
- determining a first frame portion of a first frame to be predicted, the first frame being in a video sequence;
- determining a first reference frame from the video sequence for forward inter prediction of the first frame;
- determining a second reference frame from the video sequence for backward inter prediction of the first frame;
- generating an optical flow reference frame portion for inter prediction of the first frame portion by performing an optical flow estimation using the first reference frame and the second reference frame;
- and
- performing a prediction process for the first frame portion using the optical flow reference frame.
3 more elements
An apparatus, comprising:
- a processor;
- and
- a non-transitory storage medium that includes instructions executable by the processor to carry out a method comprising:
- determining a first frame to be predicted in a video sequence;
- determining an availability of a first reference frame for forward inter prediction of the first frame and a second reference frame for backward inter prediction of the first frame;
- responsive to determining the availability of both the first reference frame and the second reference frame:
- generating a respective motion field for pixels of a first frame portion using the first reference frame and the second reference frame as input into an optical flow estimation process;
- warping a first reference frame portion to the first frame portion using the motion fields to form a first warped reference frame portion, the first reference frame portion comprising pixels of the first reference frame co-located with the pixels of the first frame portion;
- warping a second reference frame portion to the first frame portion using the motion fields to form a second warped reference frame portion, the second reference frame portion comprising pixels of the second reference frame co-located with the pixels of the first frame portion;
- and
- blending the first warped reference frame portion and the second warped reference frame portion to form an optical flow reference frame portion for inter prediction of a block of the first frame.
8 more elements
An apparatus, comprising:
- a processor;
- and
- a non-transitory storage medium that includes instructions executable by the processor to carry out a method comprising:
- generating an optical flow reference frame portion for inter prediction of a block of a first frame of a video sequence using a first reference frame from the video sequence and a second reference frame of the video sequence by:
- initializing motion fields for pixels of a first frame portion in a first processing level for an optical flow estimation, the first processing level representing downscaled motion within the first frame portion and comprising one level of multiple levels;
- for each level of the multiple levels:
- warping a first reference frame portion to the first frame portion using the motion fields to form a first warped reference frame portion;
- warping the second reference frame portion to the first frame portion using the motion fields to form a second warped reference frame portion;
- estimating motion fields between the first warped reference frame portion and the second warped reference frame portion using the optical flow estimation;
- and
- updating the motion fields for pixels of the first frame portion using the motion fields between the first warped reference frame portion and the second warped reference frame portion;
- for a final level of the multiple levels:
- warping the first reference frame portion to the first frame portion using the updated motion fields to form a final first warped reference frame portion;
- warping the second reference frame portion to the first frame portion using the updated motion fields to form a final second warped reference frame portion;
- and
- blending the final first warped reference frame portion and the second warped reference frame portion to form the optical flow reference frame portion.
13 more elements
Dependencies are inferred from the claim text.