# US 12,015,772 B2 · Prediction refinement for affine merge and affine motion vector prediction mode

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,015,772 B2
- Document ID: US12015772B2
- Status: Granted
- Kind: B2
- Application number: 17/950,443
- Filed: 2022-09-22
- Published: 2023-02-09
- Granted: 2024-06-18
- Earliest claimed: 2020-03-23
- Assignees: Beijing Bytedance Network Technology Co., Ltd.; Bytedance Inc.
- Applicants: Beijing Bytedance Network Technology Co., Ltd.; Bytedance Inc.
- Inventors: Zhipin Deng; Li Zhang; Ye-Kui Wang; Kai Zhang; Jizheng Xu
- Web page: https://patentgrep.com/patent/US12015772B2

## CPC

- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
- H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
- H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/186: the unit being a colour or a chrominance component
- H04N 19/463: by compressing encoding parameters before transmission
- H04N 19/52: by predictive encoding
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method includes determining, for a conversion between a video block of a video and a bitstream of the video, a size of prediction block corresponding to the video block according to a rule. The method also includes performing the conversion based on the determining. The rule specifies that a first size of the prediction block is determined responsive to whether a prediction refinement using optical flow technique is used for coding the video block. The video block has a second size and is coded using an affine merge mode or an affine advanced motion vector prediction mode.

## Claims

### Claim 1

A method of processing video data, comprising:
determining, for a conversion between a current video block of a video and a bitstream of the video, a size of a prediction block corresponding to the current video block according to a rule; and
performing the conversion based on the determining,
wherein an affine merge mode is enabled for the current video block, and
wherein the rule specifies that a first size of the prediction block is determined responsive to whether a prediction refinement using optical flow technique is enabled for the current video block, and wherein the current video block has a second size;
wherein a second width and a second height of the second size of the current video block are indicated by M and N, respectively, and M and N are integers greater than or equal to 0;
wherein a prediction sample of the prediction block is present as predSamplesLX[xL][yL],
wherein xL is between 0 and M+1 inclusively, and yL is between 0 and N+1 inclusively,
wherein the prediction sample predSamplesLX[xL][yL] is derived by invoking a luma integer sample fetching process for one or more of conditions are true: xL is equal to 0, xL is equal to M+1, yL is equal to 0 and yL is equal to N+1, and the prediction sample predSamplesLX[xL][yL] is derived by invoking a luma sample 8-tap interpolation filtering process for all conditions are false.

### Claim 2 (depends on 1)

The method of claim 1, wherein a first width and a first height of the first size of the prediction block are indicated by (M+M0) and (N+N0), respectively, and
wherein M0 and N0 are integers greater than or equal to 0.

### Claim 3 (depends on 2)

The method of claim 2, wherein in a case that the prediction refinement using optical flow technique is enabled for the current video block, at least one of M0 and N0 is not equal to 0.

### Claim 4 (depends on 3)

The method of claim 3, wherein M0 and N0 are both equal to 2.

### Claim 5 (depends on 2)

The method of claim 2, wherein a prediction refinement utility flag controls values of M0 and N0, and wherein the prediction refinement utility flag indicates whether the prediction refinement using optical flow technique is utilized.

### Claim 6 (depends on 5)

The method of claim 5, wherein the values of M0 and N0 are determined independently from an affine flag.

### Claim 7 (depends on 6)

The method of claim 6, wherein the affine flag is inter_affine_flag which is used to indicate whether to apply an affine motion vector prediction mode.

### Claim 8 (depends on 7)

The method of claim 7, wherein for a video block applied with the affine motion vector prediction mode, a size of a prediction block of the video block is equal to the first size.

### Claim 9 (depends on 1)

The method of claim 1, wherein the affine merge mode includes generating control point motion vectors by using a merge index to select an affine merge candidate from a sub-block merge candidate list which is constructed based on motion information of spatial neighboring coding units.

### Claim 10 (depends on 1)

The method of claim 1, wherein performing the conversion comprises encoding the video into the bitstream.

### Claim 11 (depends on 1)

The method of claim 1, wherein performing the conversion comprises decoding the video from the bitstream.

### Claim 12

An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:
determine, for a conversion between a current video block of a video and a bitstream of the video, a size of a prediction block corresponding to the current video block according to a rule; and
perform the conversion based on the determining,
wherein an affine merge mode is enabled for the current video block,
wherein the rule specifies that a first size of the prediction block is determined responsive to whether a prediction refinement using optical flow technique is enabled for the current video block, and
wherein the current video block has a second size;
wherein a second width and a second height of the second size of the current video block are indicated by M and N, respectively, and M and N are integers greater than or equal to 0,
wherein a prediction sample of the prediction block is present as predSamplesLX[xL][yL],
wherein xL is between 0 and M+1 inclusively, and yL is between 0 and N+1 inclusively, and
wherein the prediction sample predSamplesLX[xL][yL] is derived by invoking a luma integer sample fetching process for one or more of conditions are true: xL is equal to 0, xL is equal to M+1, yL is equal to 0 and yL is equal to N+1, and the prediction sample predSamplesLX[xL][yL] is derived by invoking a luma sample 8-tap interpolation filtering process for all conditions are false.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein a first width and a first height of the first size of the prediction block are indicated by (M+M0) and (N+N0), respectively,
wherein M0 and N0 are integers greater than or equal to 0,
wherein in a case that the prediction refinement using optical flow technique is enabled for the current video block, at least one of M0 and N0 is not equal to 0, and
wherein M0 and N0 are both equal to 2.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein a prediction refinement utility flag controls values of M0 and N0,
wherein the prediction refinement utility flag indicates whether the prediction refinement using optical flow technique is utilized,
wherein the values of M0 and N0 are determined independently from an affine flag,
wherein the affine flag is inter_affine_flag which is used to indicate whether to apply an affine motion vector prediction mode,
wherein for a video block applied with the affine motion vector prediction mode, a size of a prediction block of the video block is equal to the first size, and
wherein the affine merge mode includes generating control point motion vectors by using a merge index to select an affine merge candidate from a sub-block merge candidate list which is constructed based on motion information of spatial neighboring coding units.

### Claim 15

A non-transitory computer-readable storage medium storing instructions that cause a processor to:
determine, for a conversion between a current video block of a video and a bitstream of the video, a size of a prediction block corresponding to the current video block according to a rule; and
perform the conversion based on the determining,
wherein an affine merge mode is enabled for the current video block, and
wherein the rule specifies that a first size of the prediction block is determined responsive to whether a prediction refinement using optical flow technique is enabled for the current video block, and
wherein the current video block has a second size;
wherein a second width and a second height of the second size of the current video block are indicated by M and N, respectively, and M and N are integers greater than or equal to 0,
wherein a prediction sample of the prediction block is present as predSamplesLX[xL][yL],
wherein xL is between 0 and M+1 inclusively, and yL is between 0 and N+1 inclusively,
wherein the prediction sample predSamplesLX[xL][yL] is derived by invoking a luma integer sample fetching process for one or more of conditions are true: xL is equal to 0, xL is equal to M+1, yL is equal to 0 and yL is equal to N+1, and the prediction sample predSamplesLX[xL][yL] is derived by invoking a luma sample 8-tap interpolation filtering process for all conditions are false.

### Claim 16 (depends on 15)

The non-transitory computer-readable storage medium of claim 15, wherein a first width and a first height of the first size of the prediction block are indicated by (M+M0) and (N+N0), respectively,
wherein M0 and N0 are integers greater than or equal to 0,
wherein in a case that the prediction refinement using optical flow technique is enabled for the current video block, at least one of M0 and N0 is not equal to 0, and
wherein M0 and N0 are both equal to 2.

### Claim 17 (depends on 16)

The non-transitory computer-readable storage medium of claim 16, wherein a prediction refinement utility flag controls values of M0 and N0,
wherein the prediction refinement utility flag indicates whether the prediction refinement using optical flow technique is utilized,
wherein the values of M0 and N0 are determined independently from an affine flag,
wherein the affine flag is inter_affine_flag which is used to indicate whether to apply an affine motion vector prediction mode,
wherein for a video block applied with the affine motion vector prediction mode, a size of a prediction block of the video block is equal to the first size, and
wherein the affine merge mode includes generating control point motion vectors by using a merge index to select an affine merge candidate from a sub-block merge candidate list which is constructed based on motion information of spatial neighboring coding units.

### Claim 18

A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises:
determining, for a current video block of the video, a size of a prediction block corresponding to the current video block according to a rule; and
generating the bitstream based on the determining,
wherein an affine merge mode is enabled for the current video block, and
wherein the rule specifies that a first size of the prediction block is determined responsive to whether a prediction refinement using optical flow technique is enabled for the current video block, and
wherein the current video block has a second size;
wherein a second width and a second height of the second size of the current video block are indicated by M and N, respectively, and M and N are integers greater than or equal to 0,
wherein a prediction sample of the prediction block is present as predSamplesLX[xL][yL],
wherein xL is between 0 and M+1 inclusively, and yL is between 0 and N+1 inclusively, and
wherein the prediction sample predSamplesLX[xL][yL] is derived by invoking a luma integer sample fetching process for one or more of conditions are true: xL is equal to 0, xL is equal to M+1, yL is equal to 0 and yL is equal to N+1, and the prediction sample predSamplesLX[xL][yL] is derived by invoking a luma sample 8-tap interpolation filtering process for all conditions are false.

### Claim 19 (depends on 18)

The non-transitory computer-readable recording medium of claim 18, wherein a first width and a first height of the first size of the prediction block are indicated by (M+M0) and (N+N0), respectively,
wherein M0 and N0 are integers greater than or equal to 0,
wherein in a case that the prediction refinement using optical flow technique is enabled for the current video block, at least one of M0 and N0 is not equal to 0,
wherein M0 and N0 are both equal to 2,
wherein a prediction refinement utility flag controls values of M0 and N0,
wherein the prediction refinement utility flag indicates whether the prediction refinement using optical flow technique is utilized,
wherein the values of M0 and N0 are determined independently from an affine flag,
wherein the affine flag is inter_affine_flag which is used to indicate whether to apply an affine motion vector prediction mode,
wherein for a video block applied with the affine motion vector prediction mode, a size of a prediction block of the video block is equal to the first size, and
wherein the affine merge mode includes generating control point motion vectors by using a merge index to select an affine merge candidate from a sub-block merge candidate list which is constructed based on motion information of spatial neighboring coding units.