Gradient computation in bi-directional optical flow

A video processing apparatus comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to implement:

• perform, in a motion vector refinement process, for a temporal gradient of a first block or sub-block with a first size, a temporal gradient modification process to generate modified temporal gradients of the first block or sub-block;
• and
• perform the motion vectors vector refinement process for a second block or sub-block with a second size according to the modified temporal gradients of the first block or sub-block, wherein the first size is different from the second size;
• wherein, in the temporal gradient modification process, a temporal gradient is modified by removing a mean difference between a reference block 0 and a reference block 1 for the first block or sub-block is removed, and
• wherein the first block or sub-block comprises M1×N1 samples, and the second block or sub-block comprises M2×N2 samples, where M1, N1, M2 and N2 are integers, and where M1=N1=8 or M1=N1=4, and where M2 =N2 =8 or M2 =N2 =4, and
• derive refined motion vectors for a third block or sub-blocks within the third block by using partial temporal and/or spatial gradients that are calculated using partial pixels of the third block or sub-blocks within the third block such that pixels from every N rows and/or columns are used to calculate the temporal and/or spatial gradients, where N is an integer.

A computer program product stored on a non-transitory computer readable media, the computer program product including program code for carrying out a method, comprising:

• performing, in a motion vector refinement process, for a temporal gradient of a first block or sub-block with a first size, a temporal gradient modification process to generate modified temporal gradients of the first block or sub-block;
• and
• performing the motion vector refinement process for a second block or sub-block with a second size according to the modified temporal gradients of the first block or sub-block, wherein the first size is different from the second size;
• wherein, in the temporal gradient modification process, a temporal gradient is modified by removing a mean difference between a reference block 0 and a reference block 1 for the first block or sub-block, and
• wherein the first block or sub-block comprises M1×N1 samples, and the second block or sub-block comprises M2×N2 samples, where M1, N1, M2 and N2 are integers, and where M1=N1=8 or M1=N1=4, and where M2 =N2 =8 or M2 =N2 =4, and
• deriving refined motion vectors for a third block or sub-blocks within the third block by using partial temporal and/or spatial gradients that are calculated using partial pixels of the third block or sub-blocks within the third block such that pixels from every N rows and/or columns are used to calculate the temporal and/or spatial gradients, where N is an integer.