# US 2020/0267391 A1 · BLOCK-BASED Optical Flow Estimation FOR MOTION COMPENSATED PREDICTION IN VIDEO CODING

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0267391 A1
- Document ID: US20200267391A1
- Status: Published application
- Kind: A1
- Application number: 16/866,591
- Filed: 2020-05-05
- Published: 2020-08-20
- Earliest claimed: 2017-11-20
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Yaowu Xu; Bohan Li; Jingning Han
- Web page: https://patentgrep.com/patent/US20200267391A1

## CPC

- H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/172: the region being a picture, frame or field
- H04N 19/537: Motion estimation other than block-based
- H04N 19/573: Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
- H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Decoding a current block of a current frame includes selecting a first reference frame for forward inter prediction of the current frame; selecting a second reference frame for backward inter prediction of the current frame; generating an optical flow reference frame portion for inter prediction of the current block by performing an optical flow estimation using the first reference frame and the second reference frame, where the optical flow estimation produces a respective motion field for pixels of the current block; and performing a prediction process for the current block using the optical flow reference frame portion by: using a motion vector used to encode the current block to identify a reference block; adjusting boundaries of the reference block using a subpixel interpolation filter length; and identifying blocks encompassing pixels within the adjusted boundaries of the reference block.

## Claims

### Claim 1

A method for decoding a current block of a current frame, comprising:
selecting a first reference frame from a video sequence for forward inter prediction of the current frame;
selecting a second reference frame from the video sequence for backward inter prediction of the current frame, wherein the second reference frame is different from the first reference frame;
generating an optical flow reference frame portion for inter prediction of the current block by performing an optical flow estimation using the first reference frame and the second reference frame, wherein the optical flow estimation produces a respective motion field for pixels of the current block; and
performing a prediction process for the current block using the optical flow reference frame portion by:
using a motion vector used to encode the current block to identify a reference block;
adjusting boundaries of the reference block using a subpixel interpolation filter length; and
identifying blocks encompassing pixels within the adjusted boundaries of the reference block.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the optical flow reference frame portion comprises:
performing the optical flow estimation by minimizing a Lagrangian function for respective pixels of the current block.

### Claim 3 (depends on 1)

The method of claim 1, wherein the optical flow estimation produces a respective motion field for pixels of the current block, and generating the optical flow reference frame portion comprises:
warping pixels of the first reference frame that are co-located with the current block to the current block using the motion fields to form a first warped reference frame portion;
warping pixels of the second reference frame that are co-located with the current block to the current block using the motion fields to form a second warped reference frame portion; and
blending the first warped reference frame portion and the second warped reference frame portion to form the optical flow reference frame portion.

### Claim 4 (depends on 3)

The method of claim 3, wherein blending the first warped reference frame portion and the second warped reference frame portion comprises:
combining co-located pixel values of the first warped reference frame portion and the second warped reference frame portion by scaling the co-located pixel values using distances between the first reference frame and the second reference frame and between the current frame and each of the first reference frame and the second reference frame.

### Claim 5 (depends on 3)

The method of claim 3, wherein blending the first warped reference frame portion and the second warped reference frame portion comprises:
populating pixel positions of the optical flow reference frame portion by one of combining co-located pixel values of the first warped reference frame portion and the second warped reference frame portion or using a single pixel value of one or the first warped reference frame portion or the second warped reference frame portion.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
decoding, from a compressed bitstream, an indication that the current block is to be decoded using the optical flow reference frame portion.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first reference frame is a nearest reconstructed frame in a display order of the video sequence to the current frame that is available for forward inter prediction of the current frame, and the second reference frame is a nearest reconstructed frame in the display order to the current frame that is available for backward inter prediction of the current frame.

### Claim 8 (depends on 1)

The method of claim 1, wherein the optical flow reference frame portion is used as a single reference frame.

### Claim 9

An apparatus, comprising:
a processor; and
a non-transitory storage medium that includes instructions executable by the processor to carry out a method comprising:
determining a current frame to be predicted in a video sequence;
determining an availability of a first reference frame for forward inter prediction of the current frame and a second reference frame for backward inter prediction of the current frame;
responsive to determining the availability of both the first reference frame and the second reference frame:
generating a respective motion field for pixels of a current block using the first reference frame and the second reference frame as input into an optical flow estimation process;
warping a first reference frame portion to the current block using the motion fields to form a first warped reference frame portion, the first reference frame portion comprising pixels of the first reference frame that are co-located with the pixels of the current block;
warping a second reference frame portion to the current block using the motion fields to form a second warped reference frame portion, the second reference frame portion comprising pixels of the second reference frame that are co-located with the pixels of the current block; and
blending the first warped reference frame portion and the second warped reference frame portion to form an optical flow reference frame portion for inter prediction of a block of the current frame.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the method further comprises:
performing a prediction process for the block of the current frame using the optical flow reference frame portion.

### Claim 11 (depends on 9)

The apparatus of claim 9, wherein the method further comprises:
using the optical flow reference frame portion only for single reference inter prediction of blocks of the current frame.

### Claim 12 (depends on 9)

The apparatus of claim 9, wherein generating a respective motion field comprises:
calculating an output of a Lagrangian function for respective pixels of the current block using the first reference frame portion and the second reference frame portion.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein calculating the output of the Lagrangian function comprises:
calculating a first set of motion fields for the pixels of the current block using a first value for a Lagrangian parameter; and
using the first set of motion fields as input to the Lagrangian function using a second value for the Lagrangian parameter to calculate a refined set of motion fields for the pixels of the current block, wherein the second value for the Lagrangian parameter is smaller than the first value for the Lagrangian parameter, and the first warped reference frame portion and the second warped reference frame portion are warped using the refined set of motion fields.

### Claim 14

An apparatus for encoding a current block of a current frame, comprising:
a memory; and
a processor, the processor configured to execute instructions stored in the memory to:
select a first reference frame from a video sequence for forward inter prediction of the current frame;
select a second reference frame from the video sequence for backward inter prediction of the current frame, wherein the second reference frame is different from the first reference frame;
generate an optical flow reference frame portion for inter prediction of the current block by performing an optical flow estimation using the first reference frame and the second reference frame, wherein the optical flow estimation produces a respective motion field for pixels of the current block; and
perform a prediction process for the current block using the optical flow reference frame portion by instructions to:
use a motion vector used to encode the current block to identify a reference block;
adjust boundaries of the reference block using a subpixel interpolation filter length; and
identify blocks encompassing pixels within the adjusted boundaries of the reference block.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein to generate the optical flow reference frame portion comprises to:
perform the optical flow estimation by minimizing a Lagrangian function for respective pixels of the current block.

### Claim 16 (depends on 14)

The apparatus of claim 14, wherein the optical flow estimation produces a respective motion field for pixels of the current block, and wherein to generate the optical flow reference frame portion comprises to:
warp pixels of the first reference frame that are co-located with the current block to the current block using the motion fields to form a first warped reference frame portion;
warp pixels of the second reference frame that are co-located with the current block to the current block using the motion fields to form a second warped reference frame portion; and
blend the first warped reference frame portion and the second warped reference frame portion to form the optical flow reference frame portion.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein to blend the first warped reference frame portion and the second warped reference frame portion comprises to:
combine co-located pixel values of the first warped reference frame portion and the second warped reference frame portion by scaling the co-located pixel values using distances between the first reference frame and the second reference frame and between the current frame and each of the first reference frame and the second reference frame.

### Claim 18 (depends on 16)

The apparatus of claim 16, wherein to blend the first warped reference frame portion and the second warped reference frame portion comprises to:
populate pixel positions of the optical flow reference frame portion by one of combining co-located pixel values of the first warped reference frame portion and the second warped reference frame portion or using a single pixel value of one or the first warped reference frame portion or the second warped reference frame portion.

### Claim 19 (depends on 14)

The apparatus of claim 14, wherein the current block comprises one of a current block of the current frame or the current frame, and the optical flow reference frame portion is a block when the current block comprises the current block and is an entire frame when the current block comprises the current frame.

### Claim 20 (depends on 14)

The apparatus of claim 14, wherein the first reference frame is a nearest reconstructed frame in a display order of the video sequence to the current frame that is available for forward inter prediction of the current frame, and the second reference frame is a nearest reconstructed frame in the display order to the current frame that is available for backward inter prediction of the current frame.
