# US 12,225,206 B2 · Affine inter prediction refinement with optical flow

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,225,206 B2
- Document ID: US12225206B2
- Status: Granted
- Kind: B2
- Application number: 17/738,486
- Filed: 2022-05-06
- Published: 2022-08-18
- Granted: 2025-02-11
- Earliest claimed: 2019-03-18
- Assignees: Tencent America LLC
- Applicants: Tencent America LLC
- Inventors: Guichun Li; Xiang Li; Xiaozhong Xu; Shan Liu
- Web page: https://patentgrep.com/patent/US12225206B2

## CPC

- H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
- H04N 19/513: Processing of motion vectors
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/14: Coding unit complexity, e.g. amount of activity or edge presence estimation (H04N19/146 takes precedence)
- H04N 19/172: the region being a picture, frame or field
- H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/182: the unit being a pixel
- H04N 19/46: Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence)
- H04N 19/52: by predictive encoding
- H04N 19/523: with sub-pixel accuracy
- H04N 19/54: using feature points or meshes
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An apparatus for video decoding includes processing circuitry. The processing circuitry can be configured to determine an intermediate motion vector MVblock of a current block having a first precision that is higher than a second precision supported by an interpolation filter. The processing circuitry can be configured to round the intermediate motion vector MVblock to generate a rounded motion vector MVround of the current block having the second precision supported by the interpolation filter. The processing circuitry can be configured to generate a prediction block of the current block based on the rounded motion vector MVround and the interpolation filter. The processing circuitry can be configured to determine a prediction refinement offset for a prediction sample of the prediction block, and add the prediction refinement offset to the prediction sample of the prediction block to generate a refined prediction sample.

## Claims

### Claim 1

A method of video decoding by a video decoder, comprising:
determining an intermediate motion vector MVblock of a current block having a first precision that is higher than a second precision supported by an interpolation filter;
rounding the intermediate motion vector MVblock to generate a rounded motion vector MVround of the current block having the second precision supported by the interpolation filter;
generating a prediction block of the current block based on the rounded motion vector MVround and the interpolation filter;
determining a prediction refinement offset for a prediction sample of the prediction block based on gradients of the prediction sample of the prediction block and motion vector differences between the intermediate motion vector MVblock and the rounded motion vector MVround; and
adding the prediction refinement offset to the prediction sample of the prediction block to generate a refined prediction sample.

### Claim 2 (depends on 1)

The method of claim 1, wherein the gradients of the prediction sample of the prediction block are calculated with a 3-tap filter [−1, 0, 1].

### Claim 3 (depends on 1)

The method of claim 1, wherein the determining the prediction refinement offset comprises:
determining a prediction refinement offset ΔI (i,j) according to:
ΔI(i,j)=gx(i,j)*Δvx+gy(i,j)*Δvy,
where gx(i,j) and gy(i,j) are gradients of the prediction sample of the prediction block at a location (i,j) along x and y directions, respectively, and Δvx, and Δvy are motion vector differences between the intermediate motion vector MVblock and the rounded motion vector MVround along x and y directions, respectively.

### Claim 4 (depends on 1)

The method of claim 1, wherein the current block is a block or a sub-block coded with a temporal motion vector prediction mode (TMVP), a sub-block-based TMVP mode (SbTMVP), merge with motion vector difference (MMVD), affine advance motion vector prediction (AMVP), or a spatial motion vector prediction mode where the intermediate motion vector MVblock is generated in a motion vector scaling operation.

### Claim 5 (depends on 1)

The method of claim 1, wherein the current block is coded using a motion vector predictor with the first precision.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
receiving a syntax element indicating the first precision in a sequence parameter set (SPS), picture parameter set (PPS), a slice header, a tile group header.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
storing the intermediate motion vector MVblock having the first precision in a memory; and
coding another block in a current picture including the current block using the intermediate motion vector MVblock for spatial motion vector prediction.

### Claim 8

A method of video encoding by a video encoder, comprising:
determining an intermediate motion vector MVblock of a current block having a first precision that is higher than a second precision supported by an interpolation filter;
rounding the intermediate motion vector MVblock to generate a rounded motion vector MVround of the current block having the second precision supported by the interpolation filter;
obtaining a prediction block of the current block based on the rounded motion vector MVround and the interpolation filter;
determining a prediction refinement offset for a prediction sample of the prediction block based on gradients of the prediction sample of the prediction block and motion vector differences between the intermediate motion vector MVblock and the rounded motion vector MVround; and
adding the prediction refinement offset to the prediction sample of the prediction block to generate a refined prediction sample.

### Claim 9 (depends on 8)

The method of claim 8, wherein the gradients of the prediction sample of the prediction block are calculated with a 3-tap filter [−1, 0, 1].

### Claim 10 (depends on 8)

The method of claim 8, wherein the determining the prediction refinement offset comprises:
determining a prediction refinement offset ΔI (i,j) according to:
ΔI(i,j)=gx(i,j)*Δvx+gy(i,j)*Δvy,
where gx(i,j) and gy(i,j) are gradients of the prediction sample of the prediction block at a location (i,j) along x and y directions, respectively, and Δvx, and Δvy are motion vector differences between the intermediate motion vector MVblock and the rounded motion vector MVround along x and y directions, respectively.

### Claim 11 (depends on 8)

The method of claim 8, wherein the current block is a block or a sub-block to be coded with a temporal motion vector prediction mode (TMVP), a sub-block-based TMVP mode (SbTMVP), merge with motion vector difference (MMVD), affine advance motion vector prediction (AMVP), or a spatial motion vector prediction mode where the intermediate motion vector MVblock is generated in a motion vector scaling operation.

### Claim 12 (depends on 8)

The method of claim 8, further comprising:
encoding the current block in a bitstream using a motion vector predictor with the first precision.

### Claim 13 (depends on 8)

The method of claim 8, further comprising:
encoding a syntax element indicating the first precision in a sequence parameter set (SPS), picture parameter set (PPS), a slice header, a tile group header in a bitstream.

### Claim 14 (depends on 8)

The method of claim 8, further comprising:
storing the intermediate motion vector MVblock having the first precision in a memory; and
encoding another block in a current picture including the current block using the intermediate motion vector MVblock for spatial motion vector prediction.

### Claim 15

A method of processing visual media data, the method comprising:
processing a bitstream that includes the visual media data according to a format rule, wherein
the bitstream includes prediction information of a current block; and
the format rule specifies that
when an intermediate motion vector MVblock of the current block has a first precision that is higher than a second precision supported by an interpolation filter, the intermediate motion vector MVblock is rounded to generate a rounded motion vector MVround of the current block having the second precision supported by the interpolation filter,
a prediction block of the current block is generated based on the rounded motion vector MVround and the interpolation filter,
a prediction refinement offset for a prediction sample of the prediction block is determined based on gradients of the prediction sample of the prediction block and motion vector differences between the intermediate motion vector MVblock and the rounded motion vector MVround, and
the prediction refinement offset is added to the prediction sample of the prediction block to generate a refined prediction sample.

### Claim 16 (depends on 15)

The method of claim 15, wherein the gradients of the prediction sample of the prediction block are calculated with a 3-tap filter [−1, 0, 1].

### Claim 17 (depends on 15)

The method of claim 15, wherein the format rule specifies that:
a prediction refinement offset ΔI (i,j) is determined according to:
ΔI(i,j)=gx(i,j)*Δvx+gy(i,j)*Δvy,
where gx(i,j) and gy(i,j) are gradients of the prediction sample of the prediction block at a location (i,j) along x and y directions, respectively, and Δvx, and Δvy are motion vector differences between the intermediate motion vector MVblock and the rounded motion vector MVround along x and y directions, respectively.

### Claim 18 (depends on 15)

The method of claim 15, wherein the current block is a block or a sub-block coded with a temporal motion vector prediction mode (TMVP), a sub-block-based TMVP mode (SbTMVP), merge with motion vector difference (MMVD), affine advance motion vector prediction (AMVP), or a spatial motion vector prediction mode where the intermediate motion vector MVblock is generated in a motion vector scaling operation.

### Claim 19 (depends on 15)

The method of claim 15, wherein the current block is coded using a motion vector predictor with the first precision.

### Claim 20 (depends on 15)

The method of claim 15, wherein the bitstream includes a syntax element indicating the first precision in a sequence parameter set (SPS), picture parameter set (PPS), a slice header, a tile group header.