# US 10,375,413 B2 · Bi-directional optical flow for video coding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,375,413 B2
- Document ID: US10375413B2
- Status: Granted
- Kind: B2
- Application number: 15/277,433
- Filed: 2016-09-27
- Published: 2017-03-30
- Granted: 2019-08-06
- Earliest claimed: 2015-09-28
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Xiang Li; Jianle Chen; Wei-Jung Chien; Marta Karczewicz
- Web page: https://patentgrep.com/patent/US10375413B2

## CPC

- H04N 19/52: by predictive encoding
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
- H04N 19/182: the unit being a pixel
- H04N 19/573: Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
- H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An example device for decoding video data includes a processor configured to decode first and second reference pictures, wherein the first reference picture and the second reference picture are either both to be displayed before or after the current picture, determine a first motion vector (MV0) from a current block of the current picture to a first reference block of the first reference picture, determine a second motion vector (MV1) from the current block to a second reference block of the second reference picture, determine a first temporal distance (TD0) from the current picture to the first reference picture, determine a second temporal distance (TD1) from the current picture to the second reference picture, and decode the current block according to bi-directional optical flow (BIO) using MV0, MV1, TD0, and TD1.

## Claims

### Claim 1

A method of decoding video data, the method comprising:
decoding a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture and the second reference picture are either both to be displayed before a current picture or both to be displayed after the current picture;
determining a first motion vector (MV0) from a current block of the current picture to a first reference block of the first reference picture;
determining a second motion vector (MV1) from the current block to a second reference block of the second reference picture;
determining a first temporal distance (TD0) from the current picture to the first reference picture;
determining a second temporal distance (TD1) from the current picture to the second reference picture; and
decoding the current block according to bi-directional optical flow (BIO) using MV0, MV1, TD0, and TD1, wherein decoding the current block according to BIO comprises decoding the current block according to BIO when MV0 and MV1 are in proportion to TD0 and TD1, wherein MV0 includes an x-component (MV0x) and a y-component (MV0y), wherein MV1 includes an x-component (MV1x) and a y-component (MV1y), further comprising determining that MV0 and MV1 are in proportion to TD0 and TD1 when (|MV0x* TD1−MV1x*TD0|==0 &&|MV0y*TD1−MV1y*TD0|==0).

### Claim 2 (depends on 1)

The method of claim 1, further comprising decoding a second block according to BIO when MV0′ and MV1′ of the second block are almost in proportion to TD0′ and TD1′ of the second block, wherein MV0′ includes an x-component (MV0x′) and a y-component (MV0y′), wherein MV1′ includes an x-component (MV1x′) and a y-component (MV1y′), further comprising:
determining a threshold value (Th); and
determining that MV0′ and MV1′ are almost in proportion to TD0′ and TD1′ when (|MV0x′*TD1′−MV1x′*TD0′|<=Th &&|MV0y′*TD1′−MV1y′*TD0′|<=Th).

### Claim 3 (depends on 2)

The method of claim 2, wherein determining Th comprises retrieving configuration data defining Th.

### Claim 4 (depends on 2)

The method of claim 2, wherein determining Th comprises decoding a syntax element defining Th.

### Claim 5 (depends on 4)

The method of claim 4, wherein decoding the syntax element comprises decoding at least one of a sequence parameter set (SPS) including the syntax element, a picture parameter set (PPS) including the syntax element, a slice header including the syntax element, or block-level syntax including the syntax element.

### Claim 6 (depends on 1)

The method of claim 1, wherein decoding the current block according to BIO comprises decoding the current block according to BIO only when MV0 and MV1 are non-zero.

### Claim 7 (depends on 1)

The method of claim 1, wherein decoding the current block according to BIO comprises decoding the current block according to BIO only when an illumination compensation flag (ic_flag) of the current block of the current picture has a value indicating that the current block is not in a region of illumination change.

### Claim 8 (depends on 1)

The method of claim 1, wherein decoding the current block according to BIO comprises decoding the current block according to the formula:
I
t
=
I
t
⁢
⁢
0
+
I
t
⁢
⁢
1
2
+
(
G
x
⁢
⁢
0
·
V
x
⁢
⁢
0
·
TD
0
-
G
x
⁢
⁢
1
·
V
x
⁢
⁢
1
·
TD
1
)
+
(
G
y
⁢
⁢
0
·
V
y
⁢
⁢
0
·
TD
0
-
G
y
⁢
⁢
1
·
V
y
⁢
⁢
1
·
TD
1
)
2
,
wherein It0 is a pixel on a motion trajectory of MV0, It1 is a pixel on a motion trajectory of MV1, Gx0 and Gy0 are gradients for the pixel on the motion trajectory of MV0, Gx1 and Gy1 are gradients for the pixel on the motion trajectory of MV1, Vx0 and Vy0 define moving speeds for the pixel on the motion trajectory of MV0, and Vx1 and Vy1 define moving speeds for the pixel on the motion trajectory of MV1.

### Claim 9 (depends on 1)

The method of claim 1, wherein decoding the current block according to BIO comprises decoding the current block based at least in part on a gradient with respect to the first reference picture (Gx), a gradient with respect to the second reference picture (Gy), TD0, and TD1.

### Claim 10 (depends on 9)

The method of claim 9, wherein decoding the current block comprises decoding the current block using Gx′ and Gy′, wherein Gx′=Gx·TD0 and Gy′=Gy·TD1.

### Claim 11 (depends on 9)

The method of claim 9, wherein decoding the current block comprises decoding the current block using Gx′ and Gy′, wherein Gx,′ is calculated from Gx and a factor related to TD0, and Gy′ is calculated from Gy and a factor related to TD1.

### Claim 12 (depends on 1)

The method of claim 1, further comprising:
adding the first reference picture to Reference Picture List 0 for the current picture; and
adding the second reference picture to Reference Picture List 0 for the current picture.

### Claim 13 (depends on 1)

The method of claim 1, further comprising:
adding the first reference picture to Reference Picture List 1 for the current picture; and
adding the second reference picture to Reference Picture List 1 for the current picture.

### Claim 14 (depends on 1)

The method of claim 1,
wherein determining TD0 comprises calculating TD0 as being equal to a first difference between a current picture order count (POC) value for the current picture and a first POC value for the first reference picture, and
wherein determining TD1 comprises calculating TD1 as being equal to a second difference between the current POC value for the current picture and a second POC value for the second reference picture.

### Claim 15

A method of encoding video data, the method comprising:
decoding a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture and the second reference picture are either both to be displayed before a current picture or both to be displayed after the current picture;
determining a first motion vector (MV0) from a current block of the current picture to a first reference block of the first reference picture;
determining a second motion vector (MV1) from the current block to a second reference block of the second reference picture;
determining a first temporal distance (TD0) from the current picture to the first reference picture;
determining a second temporal distance (TD1) from the current picture to the second reference picture; and
encoding the current block according to bi-directional optical flow (BIO) using MV0, MV1, TD0, and TD1, wherein encoding the current block according to BIO comprises encoding the current block according to BIO when MV0 and MV1 are in proportion to TD0 and TD1, wherein MV0 includes an x-component (MV0x) and a y-component (MV0y), wherein MV1 includes an x-component (MV1x) and a y-component (MV1y), further comprising determining that MV0 and MV1 are in proportion to TD0 and TD1 when (|MV0x*TD1−MV1x*TD0|==0 &&|MV0y*TD1−MV1y*TD0|==0).

### Claim 16 (depends on 15)

The method of claim 15, further comprising encoding a second block according to BIO when MV0′ and MV1′ of the second block are almost in proportion to TD0′ and TD1′ of the second block, wherein MV0′ includes an x-component (MV0x′) and a y-component (MV0y′), wherein MV1′ includes an x-component (MV1x′) and a y-component (MV1y′), further comprising:
determining a threshold value (Th); and
determining that MV0′ and MV1′ are almost in proportion to TD0′ and TD1′ when (|MV0x′*TD1′−MV1x′*TD0′|<=Th &&|MV0y′*TD1′−MV1y′*TD0′|<=Th).

### Claim 17 (depends on 16)

The method of claim 16, further comprising encoding a value for a syntax element defining Th.

### Claim 18 (depends on 17)

The method of claim 17, wherein encoding the syntax element comprises encoding at least one of a sequence parameter set (SPS) including the syntax element, a picture parameter set (PPS) including the syntax element, a slice header including the syntax element, or block-level syntax including the syntax element.

### Claim 19 (depends on 15)

The method of claim 15, wherein encoding the current block according to BIO comprises encoding the current block according to BIO only when MV0 and MV1 are non-zero.

### Claim 20 (depends on 15)

The method of claim 15, wherein encoding the current block according to BIO comprises encoding the current block according to BIO only when the current block is not in a region of illumination change.

### Claim 21 (depends on 15)

The method of claim 15, wherein encoding the current block according to BIO comprises encoding the current block according to the formula:
I
t
=
I
t
⁢
⁢
0
+
I
t
⁢
⁢
1
2
+
(
G
x
⁢
⁢
0
·
V
x
⁢
⁢
0
·
TD
0
-
G
x
⁢
⁢
1
·
V
x
⁢
⁢
1
·
TD
1
)
+
(
G
y
⁢
⁢
0
·
V
y
⁢
⁢
0
·
TD
0
-
G
y
⁢
⁢
1
·
V
y
⁢
⁢
1
·
TD
1
)
2
,
wherein It0 is a pixel on a motion trajectory of MV0, It1 is a pixel on a motion trajectory of MV1, Gx0 and Gy0 are gradients for the pixel on the motion trajectory of MV0, Gx1 and Gy1 are gradients for the pixel on the motion trajectory of MV1, Vx0 and Vy0 define moving speeds for the pixel on the motion trajectory of MV0, and Vx1 and Vy1 define moving speeds for the pixel on the motion trajectory of MV1.

### Claim 22 (depends on 15)

The method of claim 15, wherein encoding the current block according to BIO comprises encoding the current block based at least in part on a gradient with respect to the first reference picture (Gx), a gradient with respect to the second reference picture (Gy), TD0, and TD1.

### Claim 23 (depends on 22)

The method of claim 22, wherein encoding the current block comprises encoding the current block using Gx′ and Gy′, wherein Gx′=Gx·TD0 and Gy′=Gy·TD1.

### Claim 24 (depends on 22)

The method of claim 22, wherein encoding the current block comprises encoding the current block using Gx′ and Gy′, wherein Gx′ is calculated from Gx and a factor related to TD0, and Gy′ is calculated from Gy and a factor related to TD1.

### Claim 25

A device for decoding video data, the device comprising:
a memory configured to store video data; and
one or more processors configured to:
decode a first reference picture and a second reference picture of the video data, the second reference picture being different than the first reference picture, wherein the first reference picture and the second reference picture are either both to be displayed before a current picture or both to be displayed after the current picture;
determine a first motion vector (MV0) from a current block of the current picture to a first reference block of the first reference picture;
determine a second motion vector (MV1) from the current block to a second reference block of the second reference picture;
determine a first temporal distance (TD0) from the current picture to the first reference picture;
determine a second temporal distance (TD1) from the current picture to the second reference picture; and
decode the current block according to bi-directional optical flow (BIO) using MV0, MV1, TD0, and TD1, wherein to decode the current block according to BIO, the one or more processors are configured to decode the current block according to BIO when MV0 and MV1 are in proportion to TD0 and TD1, wherein MV0 includes an x-component (MV0x) and a y-component (MV0y), wherein MV1 includes an x-component (MV1x) and a y-component (MV1y), and wherein the one or more processors are configured to determine that MV0 and MV1 are in proportion to TD0 and TD1 when (|MV0x*TD1−MV1x*TD0|==0&&|MV0y*TD1−MV1y*TD0|==0).

### Claim 26 (depends on 25)

The device of claim 25, wherein the processors are configured to decode a second block according to BIO when MV0′ and MV1′ of the second block are almost in proportion to TD0′ and TD1′ of the second block, wherein MV0′ includes an x-component (MV0x′) and a y-component (MV0y′), wherein MV1′ includes an x-component (MV1x′) and a y-component (MV1y′), and wherein the processors are further configured to:
determine a threshold value (Th); and
determine that MV0′ and MV1′ are almost in proportion to TD0′ and TD1′ when (|MV0x′*TD1′−MV1x′*TD0′|<=Th &&|MV0y′*TD1′−MV1y′*TD0′|<=Th).

### Claim 27 (depends on 25)

The device of claim 25, wherein the processors are configured to decode the current block according to BIO only when MV0 and MV1 are non-zero.

### Claim 28 (depends on 25)

The device of claim 25, wherein the processors are configured to decode the current block according to BIO only when an illumination compensation flag (ic_flag) of the current block of the current picture has a value indicating that the current block is not in a region of illumination change.

### Claim 29 (depends on 25)

The device of claim 25, wherein the processors are configured to decode the current block according to BIO using the formula:
I
t
=
I
t
⁢
⁢
0
+
I
t
⁢
⁢
1
2
+
(
G
x
⁢
⁢
0
·
V
x
⁢
⁢
0
·
TD
0
-
G
x
⁢
⁢
1
·
V
x
⁢
⁢
1
·
TD
1
)
+
(
G
y
⁢
⁢
0
·
V
y
⁢
⁢
0
·
TD
0
-
G
y
⁢
⁢
1
·
V
y
⁢
⁢
1
·
TD
1
)
2
,
wherein It0 is a pixel on a motion trajectory of MV0, It1 is a pixel on a motion trajectory of MV1, Gx0 and Gy0 are gradients for the pixel on the motion trajectory of MV0, Gx1 and Gy1 are gradients for the pixel on the motion trajectory of MV1, Vx0 and Vy0 define moving speeds for the pixel on the motion trajectory of MV0, and Vx1 and Vy1 define moving speeds for the pixel on the motion trajectory of MV1.

### Claim 30 (depends on 25)

The device of claim 25, wherein the processors are configured to decode the current block based at least in part on a gradient with respect to the first reference picture (Gx), a gradient with respect to the second reference picture (Gy), TD0, and TD1.

### Claim 31 (depends on 25)

The device of claim 25, wherein the device comprises a display configured to display a picture including the decoded current block.

### Claim 32 (depends on 25)

The device of claim 25, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

### Claim 33

A device for decoding a picture of video data, the device comprising:
means for decoding a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture and the second reference picture are either both to be displayed before a current picture or both to be displayed after the current picture;
means for determining a first motion vector (MV0) from a current block of the current picture to a first reference block of the first reference picture;
means for determining a second motion vector (MV1) from the current block to a second reference block of the second reference picture;
means for determining a first temporal distance (TD0) from the current picture to the first reference picture;
means for determining a second temporal distance (TD1) from the current picture to the second reference picture; and
means for decoding the current block according to bi-directional optical flow (BIO) using MV0, MV1, TD0, and TD1, wherein the means for decoding the current block according to BIO comprises means for decoding the current block according to BIO when MV0 and MV1 are in proportion to TD0 and TD1, wherein MV0 includes an x-component (MV0x) and a y-component (MV0y), wherein MV1 includes an x-component (MV1x) and a y-component (MV1y), further comprising means for determining that MV0 and MV1 are in proportion to TD0 and TD1 when (|MV0x*TD1−MV1x*TD0|==0&&|MV0y*TD1−MV1y*TD0|==0).

### Claim 34 (depends on 33)

The device of claim 33, further comprising means for decoding a second block according to BIO according to BIO when MV0′ and MV1′ of the second block are almost in proportion to TD0′ and TD1′ of the second block, wherein MV0′ includes an x-component (MV0x′) and a y-component (MV0y′), wherein MV1′ includes an x-component (MV1x′) and a y-component (MV1y′), further comprising:
means for determining a threshold value (Th); and
means for determining that MV0′ and MV1′ are almost in proportion to TD0′ and TD1′ when (|MV0x′*TD1′−MV1x′*TD0′|<=Th &&|MV0y′*TD1′−MV1y′*TD0′|<=Th).

### Claim 35 (depends on 33)

The device of claim 33, wherein the means for decoding the current block according to BIO comprises means for decoding the current block according to BIO only when MV0 and MV1 are non-zero.

### Claim 36 (depends on 33)

The device of claim 33, wherein the means for decoding the current block according to BIO comprises means for decoding the current block according to BIO only when an illumination compensation flag (ic_flag) of the current block of the current picture has a value indicating that the current block is not in a region of illumination change.

### Claim 37 (depends on 33)

The device of claim 33, wherein the means for decoding the current block according to BIO comprises means for decoding the current block according to the formula:
I
t
=
I
t
⁢
⁢
0
+
I
t
⁢
⁢
1
2
+
(
G
x
⁢
⁢
0
·
V
x
⁢
⁢
0
·
TD
0
-
G
x
⁢
⁢
1
·
V
x
⁢
⁢
1
·
TD
1
)
+
(
G
y
⁢
⁢
0
·
V
y
⁢
⁢
0
·
TD
0
-
G
y
⁢
⁢
1
·
V
y
⁢
⁢
1
·
TD
1
)
2
,
wherein It0 is a pixel on a motion trajectory of MV0, It1 is a pixel on a motion trajectory of MV1, Gx0 and Gy0 are gradients for the pixel on the motion trajectory of MV0, Gx1 and Gy1 are gradients for the pixel on the motion trajectory of MV1, Vx0 and Vy0 define moving speeds for the pixel on the motion trajectory of MV0, and Vx1 and Vy1 define moving speeds for the pixel on the motion trajectory of MV1.

### Claim 38 (depends on 33)

The device of claim 33, wherein the means for decoding the current block according to BIO comprises means for decoding the current block based at least in part on a gradient with respect to the first reference picture (Gx), a gradient with respect to the second reference picture (Gy), TD0, and TD1.

### Claim 39

A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a device for decoding video data to:
decode a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture and the second reference picture are either both to be displayed before a current picture or both to be displayed after the current picture;
determine a first motion vector (MV0) from a current block of the current picture to a first reference block of the first reference picture;
determine a second motion vector (MV1) from the current block to a second reference block of the second reference picture;
determine a first temporal distance (TD0) from the current picture to the first reference picture;
determine a second temporal distance (TD1) from the current picture to the second reference picture; and
decode the current block according to bi-directional optical flow (BIO) using MV0, MV1, TD0, and TD1, wherein the instructions that cause the processor to decode the current block according to BIO comprise instructions that cause the processor to decode the current block according to BIO when MV0 and MV1 are in proportion to TD0 and TD1, wherein MV0 includes an x-component (MV0x) and a y-component (MV0y), wherein MV1 includes an x-component (MV1x) and a y-component (MV1y), further comprising instructions that cause the processor to determine that MV0 and MV1 are in proportion to TD0 and TD1 when (|MV0x*TD1MV1x* TD0|==0&&|MV0y*TD1−MV1y*TD0|==0).

### Claim 40 (depends on 39)

The non-transitory computer-readable storage medium of claim 39, further comprising instructions that cause the processor to decode a second block according to BIO when MV0′ and MV1′ of the second block are almost in proportion to TD0′ and TD1′ of the second block, wherein MV0′ includes an x-component (MV0x′) and a y-component (MV0y′), wherein MV1′ includes an x-component (MV1x′) and a y-component (MV1y′), further comprising instructions that cause the processor to:
determine a threshold value (Th); and
determine that MV0′ and MV1′ are almost in proportion to TD0′ and TD1′ when (|MV0x′*TD1−MV1x′*TD0′|<=Th &&|MV0y′*TD1′−MV1y′*TD0′|<=Th).

### Claim 41 (depends on 39)

The non-transitory computer-readable storage medium of claim 39, wherein the instructions that cause the processor to decode the current block according to BIO comprise instructions that cause the processor to decode the current block according to BIO only when MV0 and MV1 are non-zero.

### Claim 42 (depends on 39)

The non-transitory computer-readable storage medium of claim 39, wherein the instructions that cause the processor to decode the current block according to BIO comprise instructions that cause the processor to decode the current block according to BIO only when an illumination compensation flag (ic_flag) of the current block of the current picture has a value indicating that the current block is not in a region of illumination change.

### Claim 43 (depends on 39)

The non-transitory computer-readable storage medium of claim 39, wherein the instructions that cause the processor to decode the current block according to BIO comprise instructions that cause the processor to decode the current block according to the formula:
I
t
=
I
t
⁢
⁢
0
+
I
t
⁢
⁢
1
2
+
(
G
x
⁢
⁢
0
·
V
x
⁢
⁢
0
·
TD
0
-
G
x
⁢
⁢
1
·
V
x
⁢
⁢
1
·
TD
1
)
+
(
G
y
⁢
⁢
0
·
V
y
⁢
⁢
0
·
TD
0
-
G
y
⁢
⁢
1
·
V
y
⁢
⁢
1
·
TD
1
)
2
,
wherein It0 is a pixel on a motion trajectory of MV0, It1 is a pixel on a motion trajectory of MV1, Gx0 and Gy0 are gradients for the pixel on the motion trajectory of MV0, Gx1 and Gy1 are gradients for the pixel on the motion trajectory of MV1, Vx0 and Vy0 define moving speeds for the pixel on the motion trajectory of MV0, and Vx1 and Vy1 define moving speeds for the pixel on the motion trajectory of MV1.

### Claim 44 (depends on 39)

The non-transitory computer-readable storage medium of claim 39, wherein the instructions that cause the processor to decode the current block according to BIO comprise instructions that cause the processor to decode the current block based at least in part on a gradient with respect to the first reference picture (Gx), a gradient with respect to the second reference picture (Gy), TD0, and TD1.