# US 2021/0344952 A1 · Bit-width control for bi-directional optical flow

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0344952 A1
- Document ID: US20210344952A1
- Status: Published application
- Kind: A1
- Application number: 17/368,468
- Filed: 2021-07-06
- Published: 2021-11-04
- Earliest claimed: 2019-01-06
- Assignees: Beijing Dajia Internet Information Technology Co., Ltd.
- Applicants: Beijing Dajia Internet Information Technology Co., Ltd.
- Inventors: Xiaoyu XIU; Yi-Wen CHEN; Xianglin WANG
- Web page: https://patentgrep.com/patent/US20210344952A1

## CPC

- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
- H04N 19/103: Selection of coding mode or of prediction mode
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/513: Processing of motion vectors
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for controlling bit-width for bi-directional optical flow (BDOF) for video coding includes decoding a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before a current picture and the second reference picture is displayed after the current picture. A motion refinement of a coding unit (CU) is calculated by minimizing a difference between a first prediction L0 and a second prediction L1. First gradient values for the first prediction Lo and second gradient values for the second prediction L1 are calculated. A final bi-prediction of the CU is calculated. Also disclosed are an associated apparatus and an associated non-transitory computer readable storage medium.

## Claims

### Claim 1

A method for video decoding, comprising:
decoding, a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before a current picture and the second reference picture is displayed after the current picture;
determining, for a current block of the current picture, a first prediction L0 corresponding to the first reference picture;
determining, for the current block, a second prediction L1 corresponding to the second reference picture;
calculating a respective motion refinement (vx, vy) of each subblock of the current block by minimizing a difference between the first prediction L0 and the second prediction L1;
calculating first gradient values
∂
I
(
0
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
0
)
∂
y
⁢
(
i
,
j
)
for the first prediction L0 and second gradient values
∂
I
(
1
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
1
)
∂
y
⁢
(
i
,
j
)
for the second prediction L1, wherein I(0)(i, j) is a prediction sample at sample location (i, j) of the first prediction L0, and I(1)(i, j) is a prediction sample at the sample location (i, j) of the second prediction L1; and
calculating a final bi-prediction of the current block based on the first prediction L0, the second prediction L1, respective motion refinements (vx, vy), the first gradient values
∂
I
(
0
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
0
)
∂
y
⁢
(
i
,
j
)
,
and the second gradient values
∂
I
(
1
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
1
)
∂
y
⁢
(
i
,
j
)
.

### Claim 2 (depends on 1)

The method of claim 1, wherein a value of (vx, vy) is calculated based on following equations:
vx=S1>0?clip3(−thBDOF,thBDOF, −((S3·2nc)>>└log2S1┘)):0vy=S5>0?clip3(−thBDOF,thBDOF,−((S6·2nc−((vxS2,m) <<ns2+vxS2,s)/2)>>└log2S5┘)):0,
wherein
S
2
,
m
=
S
2
>>
n
S
2
,
S
2
,
s
=
S
2
&
⁢
(
2
n
S
2
-
1
)
.

### Claim 3 (depends on 2)

The method of claim 2, wherein a value of S1 is calculated based on ψx(i,j) for a set of prediction samples around a respective subblock of each of the first prediction L0 and the second prediction L1, wherein a value of S2 is calculated based on ψx(i,j) and ψy(i,j) for the set of prediction samples, wherein a value of S3 is calculated based on θ(i,j) and ψx(i,j) for the set of prediction samples, wherein a value of S5 is calculated based on ψy(i,j) for the set of prediction samples, and wherein a value of S6 is calculated based on θ(i,j) and ψy(i,j) for the set of prediction samples.

### Claim 4 (depends on 2)

The method of claim 2, wherein a value of nc is set equal to 2.

### Claim 5 (depends on 3)

The method of claim 3, further comprising introducing an additional bit-shift nadj to a calculation of ψx(i,j), ψy(i,j), and θ(i,j), wherein:
ψ
x
⁡
(
i
,
j
)
=
(
∂
I
(
1
)
∂
x
⁢
(
i
,
j
)
+
∂
I
(
0
)
∂
x
⁢
(
i
,
j
)
)
⪢
(
3
-
n
adj
)
ψ
y
⁡
(
i
,
j
)
=
(
∂
I
(
1
)
∂
y
⁢
(
i
,
j
)
+
∂
I
(
0
)
∂
y
⁢
(
i
,
j
)
)
⪢
(
3
-
n
adj
)
;
⁢
and
θ
⁡
(
i
,
j
)
=
(
I
(
1
)
⁡
(
i
,
j
)
⪢
(
6
-
n
adj
)
)
-
(
I
(
0
)
⁡
(
i
,
j
)
⪢
(
6
-
n
adj
)
)
.

### Claim 6 (depends on 5)

The method of claim 5, wherein a value of nadj is set equal to 2.

### Claim 7 (depends on 5)

The method of claim 5, wherein the value of S2 is calculated based on ψx(i,j)·ψy(i,j) for the set of prediction samples, and wherein the value of S6 is calculated based on θ(i,j)·ψy(i,j) for the set of prediction samples.

### Claim 8 (depends on 1)

The method of claim 1, wherein calculating gradient value
∂
I
(
k
)
∂
x
⁢
(
i
,
j
)
,
k=0, 1 and gradient value
∂
I
(
k
)
∂
y
⁢
(
i
,
j
)
,
k=0, 1 further comprises calculating a horizontal gradient and a vertical gradient at each of a first prediction position of the first prediction L0 and a second prediction position of the second prediction L1.

### Claim 9 (depends on 8)

The method of claim 8, wherein the horizontal gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
x
⁢
(
i
,
j
)
=
(
I
(
k
)
⁡
(
i
+
1
,
j
)
-
I
(
k
)
⁡
(
i
-
1
,
j
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
;
and the vertical gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
y
⁢
(
i
,
j
)
=
(
I
(
k
)
⁡
(
i
,
j
+
1
)
-
I
(
k
)
⁡
(
i
,
j
-
1
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
.

### Claim 10 (depends on 1)

The method of claim 1, wherein calculating the final bi-prediction of the current block comprises:
calculating the final bi-prediction of the current block based further on an offset value ooffset for combining the first prediction L0 and the second prediction L1.

### Claim 11 (depends on 1)

The method of claim 1, wherein the first prediction L0 is determined based on a first motion vector (MV0) from the current block to the first reference picture, and wherein the second prediction L1 is determined based on a second motion vector (MV1) from the current block to the second reference picture.

### Claim 12

A computing device, comprising:
one or more processors;
a non-transitory storage coupled to the one or more processors; and
a plurality of programs stored in the non-transitory storage that, when executed by the one or more processors, cause the computing device to perform acts comprising:
decoding a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before a current picture and the second reference picture is displayed after the current picture;
determining, for a current Nock of the current picture, a first prediction L0 corresponding to the first reference picture;
determining for the current block, a second prediction L1 corresponding to the second reference picture;
calculating a respective motion refinement (vx, vy) of each subblock of the current block by minimizing a difference between the first prediction L0 and the second prediction L1;
calculating first gradient values
∂
I
(
0
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
0
)
∂
y
⁢
(
i
,
j
)
for the first prediction L0 and second gradient values
∂
I
(
1
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
1
)
∂
y
⁢
(
i
,
j
)
for the second prediction L1, wherein I(0)(i,j) is a prediction sample at sample location (i,j) of the first prediction L0, and I(1)(i,j) is a prediction sample at the sample location (i,j) of the second prediction L1; and
calculating a final bi-prediction of the current block based on the first prediction L0, the second prediction L1, respective motion refinements (vx, vy), the first gradient values
∂
I
(
0
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
0
)
∂
y
⁢
(
i
,
j
)
,
and the second gradient values
∂
I
(
1
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
1
)
∂
y
⁢
(
i
,
j
)
.

### Claim 13 (depends on 12)

The computing device of claim 12, wherein a value of (vx, vy) is calculated based on following equations:
vx=S1>0?clip3(−thBDOF,thBDOF, −((S3·2nc)>>└log2S1┘)):0vy=S5>0?clip3(−thBDOF,thBDOF,−((S6·2nc−((vxS2,m) <<ns2+vxS2,s)/2)>>└log2S5┘)):0,
wherein
S
2
,
m
=
S
2
>>
n
S
2
,
S
2
,
s
=
S
2
&
⁢
(
2
n
S
2
-
1
)
.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein a value of Si is calculated based on ψx(i,j) for a set of prediction samples around a respective subblock of each of the first prediction L0 and the second prediction L1, wherein a value of S2 is calculated based on ψx(i,j) and ψy(i,j) for the set of prediction samples, wherein a value of S3 is calculated based on θ(i,j) and ψx(i,j) for the set of prediction samples, wherein a value of S5 is calculated based on ψy(i,j) for the set of prediction samples, and wherein a value of S6 is calculated based on θ(i,j) and Ψy(i,j) for the set of prediction samples.

### Claim 15 (depends on 13)

The computing device of claim 13, wherein a value of nc is set equal to 2.

### Claim 16 (depends on 14)

The computing device of claim 14, further comprising introducing an additional bit-shift nadj to a calculation of ψx(i,j), ψy(i,j), and θ(i,j), wherein:
ψ
x
⁡
(
i
,
j
)
⁢
=
⁢
(
∂
I
(
1
)
∂
x
⁢
⁢
(
i
,
j
)
⁢
+
⁢
∂
I
(
0
)
∂
x
⁢
⁢
(
i
,
j
)
)
⁢
⪢
⁢
(
3
⁢
-
⁢
n
a
⁢
⁢
dj
)
ψ
y
⁡
(
i
,
j
)
⁢
=
⁢
(
∂
I
(
1
)
∂
y
⁢
⁢
(
i
,
j
)
⁢
+
⁢
∂
I
(
0
)
∂
y
⁢
⁢
(
i
,
j
)
)
⁢
⪢
⁢
(
3
⁢
-
⁢
n
adj
)
;
⁢
and
θ
⁡
(
i
,
j
)
⁢
=
⁢
(
I
(
1
)
⁡
(
i
,
j
)
⁢
⪢
⁢
(
6
⁢
-
⁢
n
adj
)
)
⁢
-
⁢
(
I
(
0
)
⁡
(
i
,
j
)
⁢
⪢
⁢
(
6
⁢
-
⁢
n
adj
)
)
.
⁢

### Claim 17 (depends on 16)

The computing device of claim 16, wherein a value of nadj is set equal to 2.

### Claim 18 (depends on 16)

The computing device of claim 16, wherein the value of S2 is calculated based on ψx(i,j)·ψy(i,j) for the set of prediction samples, and wherein the value of S6 is calculated based on θ(i,j)·ψy(i,j) for the set of prediction samples.

### Claim 19 (depends on 12)

The computing device of claim 12, wherein calculating gradient value
∂
I
(
k
)
∂
x
⁢
(
i
,
j
)
,
k=0, 1 and gradient value
∂
I
(
k
)
∂
y
⁢
(
i
,
j
)
,
k=0, 1 further comprises calculating a horizontal gradient and a vertical gradient at each of a first prediction position of the first prediction L0 and a second prediction position of the second prediction L1.

### Claim 20 (depends on 19)

The computing device of claim 19, wherein the horizontal gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
x
⁢
(
i
,
j
)
=
(
I
(
k
)
⁡
(
i
+
1
,
j
)
-
I
(
k
)
⁡
(
i
-
1
,
j
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1
;
and the vertical gradient is calculated at each of the first prediction position and the second prediction position as:
∂
I
(
k
)
∂
y
⁢
(
i
,
j
)
=
(
I
(
k
)
⁡
(
i
,
j
+
1
)
-
I
(
k
)
⁡
(
i
,
j
-
1
)
)
⪢
(
4
+
n
grad
)
,
k
=
0
,
1.

### Claim 21

A non-transitory computer readable storage medium storing a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, cause the computing device to perform acts comprising:
decoding a first reference picture and a second reference picture, the second reference picture being different than the first reference picture, wherein the first reference picture is displayed before a current picture and the second reference picture is displayed after the current picture;
determining, for a current block of the current picture, a first prediction L0 corresponding to the first reference picture;
determining, for the current block, a second prediction L1 corresponding to the second reference picture;
calculating a respective motion refinement (vx, vy) of each subblock of the current block by minimizing a difference between the first prediction L0 and the second prediction L1;
calculating first gradient values
∂
I
(
0
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
0
)
∂
y
⁢
(
i
,
j
)
for the first prediction L0 and second gradient values
∂
I
(
k
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
k
)
∂
y
⁢
(
i
,
j
)
for the second prediction L1, wherein I(0)(i, j) is a prediction sample at sample location (i,j) of the first prediction L0, and I(1)(i, j) is a prediction sample at the sample location (i,j) of the second prediction L1; and
calculating a final bi-prediction of the current block based on the first prediction L0, the second prediction L1, respective motion refinements (vx, vy), the first gradient values
∂
I
(
0
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
0
)
∂
y
⁢
(
i
,
j
)
,
and the second gradient values
∂
I
(
1
)
∂
x
⁢
(
i
,
j
)
⁢
⁢
and
⁢
⁢
∂
I
(
1
)
∂
y
⁢
(
i
,
j
)
.