# US 12,003,733 B2 · Method and apparatus for prediction refinement with optical flow for an affine coded block

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,003,733 B2
- Document ID: US12003733B2
- Status: Granted
- Kind: B2
- Application number: 17/478,697
- Filed: 2021-09-17
- Published: 2022-03-10
- Granted: 2024-06-04
- Earliest claimed: 2019-03-20
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Huanbang Chen; Haitao Yang; Jianle Chen
- Web page: https://patentgrep.com/patent/US12003733B2

## CPC

- H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
- H04N 19/119: Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks
- H04N 19/172: the region being a picture, frame or field
- H04N 19/174: the region being a slice, e.g. a line of blocks or a group of blocks
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/52: by predictive encoding
- H04N 19/521: {for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors}
- H04N 19/55: Motion estimation with spatial constraints, e.g. at image or region borders
- H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The present invention relates to an apparatus, an encoder, a decoder and corresponding methods for prediction refinement with optical flow (PROF) for an affine coded block, in which when a plurality of optical flow decision conditions are fulfilled for the affine coded block, performing a PROF process for a current sub-block of the affine coded block to obtain refined prediction sample values of the current sub-block of the affine coded block. After the sub-block based affine motion compensation is performed, a prediction sample value of the current sample of the current sub-block is refined by adding a delta prediction value. Thus, it allows for a better trade-off between coding complexity and prediction accuracy.

## Claims

### Claim 1

A method for performing a prediction refinement with optical flow (PROF) process for a current sub-block of an affine coded block implemented by a coding device, comprising:
when a plurality of constraint conditions for applying PROF are not fulfilled for the affine coded block, performing the PROF process for the current sub-block of the affine coded block to obtain refined prediction sample values of the current sub-block of the affine coded block by
performing an optical flow processing for the current sub-block of the affine coded block to obtain a delta prediction value of a current sample of the current sub-block; and
obtaining a refined prediction sample value of the current sample based on the delta prediction value of the current sample and a prediction sample value of the current sample of the current sub-block;
wherein the plurality of constraint conditions for applying PROF comprises:
first indication information indicating that PROF is disabled for a picture containing the affine coded block or indicating that the PROF is disabled for slices associated with a picture containing the affine coded block; and
second indication information indicating no partition of the affine coded block.

### Claim 2 (depends on 1)

The method according to claim 1, wherein
before the performing the PROF process for the current sub-block of the affine coded block, the method further comprises: determining that the plurality of constraint conditions for applying PROF are not fulfilled for the affine coded block.

### Claim 3 (depends on 1)

The method according to claim 1, wherein one of the plurality of constraint conditions for applying PROF is that a variable fallbackModeTriggered is set to 1.

### Claim 4 (depends on 1)

The method according to claim 1, wherein the performing the optical flow processing for the current sub-block to obtain the delta prediction value of the current sample of the current sub-block comprises:
obtaining a second prediction matrix, wherein the second prediction matrix is generated based on motion information of the current sub-block;
generating a horizontal prediction gradient matrix and a vertical prediction gradient matrix based on the second prediction matrix, wherein the horizontal prediction gradient matrix and the vertical prediction gradient matrix have a same size and a size of the second prediction matrix is greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix; and
calculating the delta prediction value of the current sample of the current sub-block based on a horizontal prediction gradient value of the current sample in the horizontal prediction gradient matrix, a vertical prediction gradient value of the current sample in the vertical prediction gradient matrix, and a difference between a motion vector of the current sample of the current sub-block and a motion vector of a center sample of the current sub-block.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the obtaining the second prediction matrix comprises:
generating a first prediction matrix based on motion information of the current sub-block, wherein elements of the first prediction matrix correspond to prediction sample values of the current sub-block, and generating the second prediction matrix based on the first prediction matrix.

### Claim 6 (depends on 4)

The method according to claim 4, wherein
an element of the second prediction matrix is represented by I1(p, q), wherein a value range of p is [−1, sbW], and a value range of q is [−1, sbH],
an element of the horizontal prediction gradient matrix is represented by X(i,j) and corresponds to sample (i,j) of the current sub-block in the affine coded block, wherein a value range of i is [0, sbW−1], and a value range of j is [0, sbH−1],
an element of the vertical prediction gradient matrix is represented by Y(i,j) and corresponds to sample (i,j) of the current sub-block in the affine coded block, wherein a value range of i is [0, sbW−1], and a value range of j is [0, sbH−1], and wherein
sbW represents a width of the current sub-block in the affine coded block, and sbH represents a height of the current sub-block in the affine coded block.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising:
before the performing the PROF process for the current sub-block of the affine coded block, performing subblock-based affine motion compensation for the current sub-block of the affine coded block to obtain prediction sample values of the current sub-block.

### Claim 8

A method for performing a prediction refinement with optical flow (PROF) process for a current sub-block of an affine coded block implemented by a coding device, comprising:
when a plurality of optical flow decision conditions are fulfilled for the affine coded block, performing the PROF process for the current sub-block of the affine coded block to obtain refined prediction sample values of the current sub-block of the affine coded block by
performing an optical flow processing for the current sub-block of the affine coded block to obtain a delta prediction value of a current sample of the current sub-block; and
obtaining a refined prediction sample value of the current sample based on the delta prediction value of the current sample and a prediction sample value of the current sample of the current sub-block;
wherein the plurality of optical flow decision conditions comprises:
first indication information indicating that PROF is enabled for a picture containing the affine coded block or indicating that PROF is enabled for slices associated with a picture containing the affine coded block; and
second indication information indicating partition is applied to the affine coded block.

### Claim 9 (depends on 8)

The method according to claim 8, further comprising:
before the performing the PROF process for the current sub-block of the affine coded block,
determining that the plurality of optical flow decision conditions are fulfilled for the affine coded block.

### Claim 10 (depends on 8)

The method according to claim 8, wherein one of the plurality of the optical flow decision conditions is that a variable fallbackModeTriggered is set to 0.

### Claim 11 (depends on 8)

The method according to claim 8, wherein the performing the optical flow processing for the current sub-block to obtain the delta prediction value of the current sample of the current sub-block comprises:
obtaining a second prediction matrix, wherein the second prediction matrix is generated based on motion information of the current sub-block;
generating a horizontal prediction gradient matrix and a vertical prediction gradient matrix based on the second prediction matrix, wherein the horizontal prediction gradient matrix and the vertical prediction gradient matrix have a same size and a size of the second prediction matrix is greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix; and
calculating the delta prediction value of the current sample of the current sub-block based on a horizontal prediction gradient value of the current sample in the horizontal prediction gradient matrix, a vertical prediction gradient value of the current sample in the vertical prediction gradient matrix, and a difference between a motion vector of the current sample of the current sub-block and a motion vector of a center sample of the sub-block.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the obtaining the second prediction matrix, comprises:
generating a first prediction matrix based on motion information of the current sub-block, wherein elements of the first prediction matrix correspond to prediction sample values of the current sub-block, and generating the second prediction matrix based on the first prediction matrix.

### Claim 13 (depends on 11)

The method according to claim 11, wherein
an element of the second prediction matrix is represented by I1(p, q), wherein a value range of p is [−1, sbW], and a value range of q is [−1, sbH],
an element of the horizontal prediction gradient matrix is represented by X(i,j) and corresponds to sample (i,j) of the current sub-block in the affine coded block, wherein a value range of i is [0, sbW−1], and a value range of j is [0, sbH−1],
an element of the vertical prediction gradient matrix is represented by Y(i,j) and corresponds to sample (i,j) of the current sub-block in the affine coded block, wherein a value range of i is [0, sbW−1], and a value range of j is [0, sbH−1], and wherein
sbW represents a width of the current sub-block in the affine coded block, and sbH represents a height of the current sub-block in the affine coded block.

### Claim 14 (depends on 8)

The method according to claim 8, further comprising:
before the performing the PROF process for the current sub-block of the affine coded block, performing subblock-based affine motion compensation for the current sub-block of the affine coded block to obtain prediction sample values of the current sub-block.

### Claim 15

An apparatus for performing a prediction refinement with optical flow (PROF) process for a current sub-block of an affine coded block, comprising:
one or more electronic circuits or processors configured to:
perform, when a plurality of constraint conditions for applying PROF are not fulfilled for the affine coded block, the PROF process for the current sub-block of the affine coded block to obtain refined prediction sample values of the current sub-block of the affine coded block by
performing an optical flow processing for the current sub-block of the affine coded block to obtain a delta prediction value of a current sample of the current sub-block; and
obtaining a refined prediction sample value of the current sample based on the delta prediction value of the current sample and a prediction sample value of the current sample of the current sub-block;
wherein the plurality of constraint conditions for applying PROF comprises:
first indication information indicating that PROF is disabled for a picture containing the affine coded block or indicating that the PROF is disabled for slices associated with a picture containing the affine coded block; and
second indication information indicating no partition of the affine coded block.

### Claim 16 (depends on 15)

The apparatus according to claim 15, wherein the one or more electronic circuits or processors are further configured to determine that the plurality of constraint conditions for applying PROF are not fulfilled for the affine coded block.

### Claim 17 (depends on 15)

The apparatus according to claim 15, wherein one of the plurality of constraint conditions for applying PROF is that a variable fallbackModeTriggered is set to 1.

### Claim 18 (depends on 15)

The apparatus according to claim 15, wherein the one or more electronic circuits or processors are further configured to:
obtain a second prediction matrix, wherein the second prediction matrix is generated based on motion information of the current sub-block;
generate a horizontal prediction gradient matrix and a vertical prediction gradient matrix based on the second prediction matrix, wherein the horizontal prediction gradient matrix and the vertical prediction gradient matrix have a same size and a size of the second prediction matrix is greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix; and
calculate the delta prediction value of the current sample of the current sub-block based on a horizontal prediction gradient value of the current sample in the horizontal prediction gradient matrix, a vertical prediction gradient value of the current sample in the vertical prediction gradient matrix, and a difference between a motion vector of the current sample of the current sub-block and a motion vector of a center sample of the current sub-block.

### Claim 19 (depends on 18)

The apparatus according to claim 18, wherein the one or more electronic circuits or processors are further configured to:
generate a first prediction matrix based on motion information of the current sub-block, wherein elements of the first prediction matrix correspond to prediction sample values of the current sub-block, and generating the second prediction matrix based on the first prediction matrix; or
generate the second prediction matrix based on the motion information of the current sub-block.

### Claim 20 (depends on 18)

The apparatus according to claim 18, wherein
an element of the second prediction matrix is represented by I1(p, q), wherein a value range of p is [−1, sbW], and a value range of q is [−1, sbH],
an element of the horizontal prediction gradient matrix is represented by X(i,j) and corresponds to sample (i,j) of the current sub-block in the affine coded block, wherein a value range of i is [0, sbW−1], and a value range of j is [0, sbH−1],
an element of the vertical prediction gradient matrix is represented by Y(i,j) and corresponds to sample (i,j) of the current sub-block in the affine coded block, wherein a value range of i is [0, sbW−1], and a value range of j is [0, sbH−1], and wherein
sbW represents a width of the current sub-block in the affine coded block, and sbH represents a height of the current sub-block in the affine coded block.

### Claim 21 (depends on 15)

The apparatus according to claim 15, wherein the one or more electronic circuits or processors are further configured to:
perform subblock-based affine motion compensation for the current sub-block of the affine coded block to obtain prediction sample values of the current sub-block.

### Claim 22

An apparatus for performing a prediction refinement with optical flow (PROF) process for a current sub-block of an affine coded block, comprising:
one or more electronic circuits or processors configured to:
perform, when a plurality of optical flow decision conditions are fulfilled for the affine coded block, the PROF process for the current sub-block of the affine coded block to obtain refined prediction sample values of the current sub-block of the affine coded block by
perform an optical flow processing for the current sub-block of the affine coded block to obtain a delta prediction value of a current sample of the current sub-block; and
obtain a refined prediction sample value of the current sample based on the delta prediction value of the current sample and a prediction sample value of the current sample of the current sub-block;
wherein the plurality of optical flow decision conditions comprises:
first indication information indicating that PROF is enabled for a picture containing the affine coded block or indicating that PROF is enabled for slices associated with a picture containing the affine coded block; and
second indication information indicating partition is applied to the affine coded block.

### Claim 23 (depends on 22)

The apparatus according to claim 22, wherein the one or more electronic circuits or processors are further configured to:
determine that the plurality of optical flow decision conditions are fulfilled for the affine coded block.

### Claim 24 (depends on 22)

The apparatus according to claim 22, wherein one of the plurality of the optical flow decision conditions is that a variable fallbackModeTriggered is set to 0.

### Claim 25 (depends on 22)

The apparatus according to claim 22, wherein the one or more electronic circuits or processors are further configured to
obtain a second prediction matrix, wherein the second prediction matrix is generated based on motion information of the current sub-block;
generate a horizontal prediction gradient matrix and a vertical prediction gradient matrix based on the second prediction matrix, wherein the horizontal prediction gradient matrix and the vertical prediction gradient matrix have a same size and a size of the second prediction matrix is greater than or equal to the size of the horizontal prediction gradient matrix and the vertical prediction gradient matrix; and
calculate the delta prediction value of the current sample of the current sub-block based on a horizontal prediction gradient value of the current sample in the horizontal prediction gradient matrix, a vertical prediction gradient value of the current sample in the vertical prediction gradient matrix, and a difference between a motion vector of the current sample of the current sub-block and a motion vector of a center sample of the sub-block.

### Claim 26 (depends on 25)

The apparatus according to claim 25, wherein the one or more electronic circuits or processors are further configured to:
generate a first prediction matrix based on motion information of the current sub-block, wherein elements of the first prediction matrix correspond to prediction sample values of the current sub-block, and generating the second prediction matrix based on the first prediction matrix.

### Claim 27 (depends on 25)

The apparatus according to claim 25, wherein
an element of the second prediction matrix is represented by I1(p, q), wherein a value range of p is [−1, sbW], and a value range of q is [−1, sbH],
an element of the horizontal prediction gradient matrix is represented by X(i,j) and corresponds to sample (i,j) of the current sub-block in the affine coded block, wherein a value range of i is [0, sbW−1], and a value range of j is [0, sbH−1],
an element of the vertical prediction gradient matrix is represented by Y(i,j) and corresponds to sample (i,j) of the current sub-block in the affine coded block, wherein a value range of i is [0, sbW−1], and a value range of j is [0, sbH−1], and wherein
sbW represents a width of the current sub-block in the affine coded block, and sbH represents a height of the current sub-block in the affine coded block.

### Claim 28 (depends on 22)

The apparatus according to claim 22, wherein the one or more electronic circuits or processors are further configured to:
perform subblock-based affine motion compensation for the current sub-block of the affine coded block to obtain prediction sample values of the current sub-block.

### Claim 29

A non-transitory computer-readable media storing computer instructions for performing a prediction refinement with optical flow (PROF) process for a current sub-block of an affine coded block, that when executed by one or more processors, cause the one or more processors to perform the operations of:
when a plurality of constraint conditions for applying PROF are not fulfilled for the affine coded block, performing the PROF process for the current sub-block of the affine coded block to obtain refined prediction sample values of the current sub-block of the affine coded block by
performing an optical flow processing for the current sub-block of the affine coded block to obtain a delta prediction value of a current sample of the current sub-block; and
obtaining a refined prediction sample value of the current sample based on the delta prediction value of the current sample and a prediction sample value of the current sample of the current sub-block;
wherein the plurality of constraint conditions for applying PROF comprises:
first indication information indicating that PROF is disabled for a picture containing the affine coded block or indicating that the PROF is disabled for slices associated with a picture containing the affine coded block; and
second indication information indicating no partition of the affine coded block.

### Claim 30

A non-transitory computer-readable media storing computer instructions for performing a prediction refinement with optical flow (PROF) process for a current sub-block of an affine coded block, that when executed by one or more processors, cause the one or more processors to perform the operations of:
when a plurality of optical flow decision conditions are fulfilled for the affine coded block, performing the PROF process for the current sub-block of the affine coded block to obtain refined prediction sample values of the current sub-block of the affine coded block by
performing an optical flow processing for the current sub-block of the affine coded block to obtain a delta prediction value of a current sample of the current sub-block; and
obtaining a refined prediction sample value of the current sample based on the delta prediction value of the current sample and a prediction sample value of the current sample of the current sub-block;
wherein the plurality of optical flow decision conditions comprises:
first indication information indicating that PROF is enabled for a picture containing the affine coded block or indicating that PROF is enabled for slices associated with a picture containing the affine coded block; and
second indication information indicating partition is applied to the affine coded block.