# US 11,445,210 B2 · Decoder side motion vector derivation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,445,210 B2
- Document ID: US11445210B2
- Status: Granted
- Kind: B2
- Application number: 17/405,179
- Filed: 2021-08-18
- Published: 2021-12-09
- Granted: 2022-09-13
- Earliest claimed: 2019-04-02
- Assignees: Beijing Bytedance Network Technology Co., Ltd.; Bytedance Inc.
- Applicants: Beijing Bytedance Network Technology Co., Ltd.; Bytedance Inc.
- Inventors: Hongbin Liu; Li Zhang; Kai Zhang; Jizheng Xu; Yue Wang
- Web page: https://patentgrep.com/patent/US11445210B2

## CPC

- H04N 19/513: Processing of motion vectors
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method for processing a video includes performing a conversion between a current block of visual media data and a corresponding coded representation of the visual media data, wherein the conversion of the current block includes determining whether a use of one or both of a bi-directional optical flow (BIO) technique or a decoder-side motion vector refinement (DMVR) technique to the current block is enabled or disabled, and wherein the determining the use of the BIO technique or the DMVR technique is based on a cost criterion associated with the current block.

## Claims

### Claim 1

A method of processing video data, comprising:
determining, for a current block of a video, an initial prediction sample;
refining the initial prediction sample, based on an optical flow refinement technology, with a prediction sample offset to acquire a final prediction sample; and
performing a conversion between the current block and a bitstream of the video based on the final prediction sample,
wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is calculated based on at least a difference between two first prediction samples from a same reference picture list, and
wherein before calculating the difference between the two first prediction samples, values of the two first prediction sample are right shifted with a first value.

### Claim 2 (depends on 1)

The method of claim 1, wherein for a sample location (x,y) in the current block, the two first prediction samples have locations (hx+1, vy) and (hx−1, vy) corresponding to the same reference picture list X or locations (hx, vy+1) and (hx, vy−1) corresponding to the same reference picture list X, and
wherein X=0 or 1, hx=Clip3(1, nCbW, x) and vy=Clip3(1, nCbH, y), nCbW and nCbH is a width and a height of the current block, and wherein Clip3 is a clipping function which is defined as:
Clip
⁢
⁢
3
⁢
(
u
,
v
,
w
)
=
{
u
;
w
<
u
v
;
w
>
v
w
;
otherwise
.

### Claim 3 (depends on 1)

The method of claim 1, wherein the prediction sample offset is determined further based on at least one temporal gradient, wherein the temporal gradient is calculated based on at least a difference between two second prediction samples from different reference picture lists, and wherein before calculating the difference between the two second prediction samples, values of the two second prediction sample are right shifted with a second value.

### Claim 4 (depends on 3)

The method of claim 3, wherein for a sample location (x,y) in the current block, the two second prediction samples have locations (hx, vy) corresponding to a reference picture list 0 and a reference picture list 1, and
wherein hx=Clip3(1, nCbW, x) and vy=Clip3(1, nCbH, y), nCbW and nCbH is a width and a height of the current block, and wherein Clip3 is a clipping function which is defined as:
Clip
⁢
⁢
3
⁢
(
u
,
v
,
w
)
=
{
u
;
w
<
u
v
;
w
>
v
w
;
otherwise
.

### Claim 5 (depends on 3)

The method of claim 3, wherein the first value is different from the second value.

### Claim 6 (depends on 1)

The method of claim 1, wherein the prediction sample offset is determined further based on at least one temporal gradient, wherein the temporal gradient is calculated based on at least a difference between two second prediction samples from different reference picture list, and wherein
a shifting rule of the difference between the two second prediction samples is same as that of the difference between the two first prediction samples, and the shifting rule includes an order of a right-shifting operation and a subtraction operation.

### Claim 7 (depends on 1)

The method of claim 1, wherein whether the optical flow refinement procedure is enabled based on a condition related with a size of the current block.

### Claim 8 (depends on 7)

The method of claim 7, wherein whether a decoder-side motion vector refinement technique is enabled for the current block is based on the same condition, wherein the decoder-side motion vector refinement technique is used to derive a refined motion information of the current block based on a cost between at least one prediction sample acquired based on at least one reference sample of reference picture list 0 and at least one prediction sample acquired based on at least one reference sample of reference picture list 1.

### Claim 9 (depends on 8)

The method of claim 8, wherein the optical flow refinement technology and the decoder-side motion vector refinement technique are enabled at least based on a height of the current block is equal to or greater than T1.

### Claim 10 (depends on 9)

The method of claim 9, wherein T1=8.

### Claim 11 (depends on 1)

The method of claim 1, wherein performing the conversion includes decoding the current block from the bitstream.

### Claim 12 (depends on 1)

The method of claim 1, wherein performing the conversion includes encoding the current block into the bitstream.

### Claim 13

An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:
determine, for a current block of a video, an initial prediction sample;
refine the initial prediction sample, based on an optical flow refinement technology, with a prediction sample offset to acquire a final prediction sample; and
perform a conversion between the current block and a bitstream of the video based on the final prediction sample,
wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is calculated based on at least a difference between two first prediction samples from a same reference picture list, and
wherein before calculating the difference between the two first prediction samples, values of the two first prediction sample are right shifted with a first value.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein for a sample location (x,y) in the current block, the two first prediction samples have locations (hx+1, vy) and (hx−1, vy) corresponding to the same reference picture list X or locations (hx, vy+1) and (hx, vy−1) corresponding to the same reference picture list X, and
wherein X=0 or 1, hx=Clip3(1, nCbW, x) and vy=Clip3(1, nCbH, y), nCbW and nCbH is a width and a height of the current block, and wherein Clip3 is a clipping function which is defined as:
Clip
⁢
⁢
3
⁢
(
u
,
v
,
w
)
=
{
u
;
w
<
u
v
;
w
>
v
w
;
otherwise
.

### Claim 15 (depends on 13)

The apparatus of claim 13, wherein the prediction sample offset is determined further based on at least one temporal gradient, wherein the temporal gradient is calculated based on at least a difference between two second prediction samples from different reference picture lists, and
wherein before calculating the difference between the two second prediction samples, values of the two second prediction sample are right shifted with a second value.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein for a sample location (x,y) in the current block, the two second prediction samples have locations (hx, vy) corresponding to a reference list picture 0 and a reference picture list 1, and
wherein hx=Clip3(1, nCbW, x) and vy=Clip3(1, nCbH, y), nCbW and nCbH is a width and a height of the current block, and wherein Clip3 is a clipping function which is defined as:
Clip
⁢
⁢
3
⁢
(
u
,
v
,
w
)
=
{
u
;
w
<
u
v
;
w
>
v
w
;
otherwise
.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein the first value is different from the second value.

### Claim 18

A non-transitory computer-readable storage medium storing instructions that cause a processor to:
determine, for a current block of a video, an initial prediction sample;
refine the initial prediction sample, based on an optical flow refinement technology, with a prediction sample offset to acquire a final prediction sample; and
perform a conversion between the current block and a bitstream of the video based on the final prediction sample,
wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is calculated based on at least a difference between two first prediction samples from a same reference picture list, and
wherein before calculating the difference between the two first prediction samples, values of the two first prediction sample are right shifted with a first value.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein for a sample location (x,y) in the current block, the two first prediction samples have locations (hx+1, vy) and (hx−1, vy) corresponding to the same reference picture list X or locations (hx, vy+1) and (hx, vy−1) corresponding to the same reference picture list X, and
wherein X=0 or 1, hx=Clip3(1, nCbW, x) and vy=Clip3(1, nCbH, y), nCbW and nCbH is a width and a height of the current block, and wherein Clip3 is a clipping function which is defined as:
Clip
⁢
⁢
3
⁢
(
u
,
v
,
w
)
=
{
u
;
w
<
u
v
;
w
>
v
w
;
otherwise
.

### Claim 20

A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises:
determining, for a current block of a video, an initial prediction sample;
refining the initial prediction sample, based on an optical flow refinement technology, with a prediction sample offset to acquire a final prediction sample; and
generating the bitstream based on the final prediction sample,
wherein the prediction sample offset is determined based on at least one spatial gradient of the initial prediction sample, wherein the spatial gradient is calculated based on at least a difference between two first prediction samples from a same reference picture list, and
wherein before calculating the difference between the two first prediction samples, values of the two first prediction sample are right shifted with a first value.