# US 12,003,703 B2 · Complexity reduction and bit-width control for bi-directional optical flow > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,703 B2 - Document ID: US12003703B2 - Status: Granted - Kind: B2 - Application number: 17/962,918 - Filed: 2022-10-10 - Published: 2023-02-16 - Granted: 2024-06-04 - Earliest claimed: 2018-09-21 - Assignees: Vid Scale, Inc. - Applicants: Vid Scale, Inc. - Inventors: Xiaoyu Xiu; Yuwen He; Yan Ye; Jiancong Luo - Web page: https://patentgrep.com/patent/US12003703B2 ## CPC - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/146: Data rate or code amount at the encoder output - H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/513: Processing of motion vectors - H04N 19/52: by predictive encoding - H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures - H04N 19/563: Motion estimation with padding, i.e. with filling of non-object values in an arbitrarily shaped picture block or region for estimation purposes - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Systems and methods are described for reducing the complexity of using bi-directional optical flow (BIO) in video coding. In some embodiments, bit-width reduction steps are introduced in the BIO motion refinement process to reduce the maximum bit-width used for BIO calculations. In some embodiments, simplified interpolation filters are used to generate predicted samples in an extended region around a current coding unit. In some embodiments, different interpolation filters are used for vertical versus horizontal interpolation. In some embodiments, BIO is disabled for coding units with small heights and/or for coding units that are predicted using a sub-block level inter prediction technique, such as advanced temporal motion vector prediction (ATMVP) or affine prediction. ## Claims ### Claim 1 A video encoding method comprising: obtaining a first prediction signal array from a first reference picture and a second prediction signal array from a second reference picture; performing a right bit shift on the first prediction signal array to obtain a bit-shifted first prediction signal array and on the second prediction signal array to obtain a bit-shifted second prediction signal array; obtaining a signal-difference parameter array based on a difference between the bit-shifted first prediction signal array and the bit-shifted second prediction signal array; obtaining at least a first-component motion refinement based at least in part on the signal-difference parameter array; and generating a prediction of a current block in a video with bi-directional optical flow using at least the first-component motion refinement. ### Claim 2 (depends on 1) The method of claim 1, further comprising: obtaining a first array of horizontal gradients based on the first prediction signal array; obtaining a second array of horizontal gradients based on a the second prediction signal array; and obtaining a horizontal intermediate parameter array including performing a right bit shift on a sum of the first array of horizontal gradients and the second array of horizontal gradients; wherein the first-component motion refinement is a horizontal motion refinement and is further based at least in part on the horizontal intermediate parameter array. ### Claim 3 (depends on 2) The method of claim 2, wherein the horizontal motion refinement is based at least in part on a product of the signal-difference parameter array and the horizontal intermediate parameter array. ### Claim 4 (depends on 1) The method of claim 1, further comprising: obtaining a first array of vertical gradients based on the first prediction signal array; obtaining a second array of vertical gradients based on a the second prediction signal array; and obtaining a vertical intermediate parameter array including performing a right bit shift on a sum of the first array of vertical gradients and the second array of vertical gradients; wherein the first-component motion refinement is a vertical motion refinement and is further based at least in part on the vertical intermediate parameter array. ### Claim 5 (depends on 2) The method of claim 2, wherein obtaining the first array of horizontal gradients comprises padding samples outside the first prediction sample array with respective nearest boundary samples inside the first prediction sample array, and obtaining the second array of horizontal gradients comprises padding samples outside the second prediction sample array with respective nearest boundary samples inside the second prediction sample array. ### Claim 6 A video encoder comprising one or more processors configured to perform: obtaining a first prediction signal array from a first reference picture and a second prediction signal array from a second reference picture; performing a right bit shift on the first prediction signal array to obtain a bit-shifted first prediction signal array and on the second prediction signal array to obtain a bit-shifted second prediction signal array; obtaining a signal-difference parameter array based on a difference between the bit-shifted first prediction signal array and the bit-shifted second prediction signal array; obtaining at least a first-component motion refinement based at least in part on the signal-difference parameter array; and generating a prediction of a current block in a video with bi-directional optical flow using at least the first-component motion refinement. ### Claim 7 (depends on 6) The video encoder of claim 6, further configured to perform: obtaining a first array of horizontal gradients based on the first prediction signal array; obtaining a second array of horizontal gradients based on a the second prediction signal array; and obtaining a horizontal intermediate parameter array including performing a right bit shift on a sum of the first array of horizontal gradients and the second array of horizontal gradients; wherein the first-component motion refinement is a horizontal motion refinement and is further based at least in part on the horizontal intermediate parameter array. ### Claim 8 (depends on 7) The video encoder of claim 7, wherein the horizontal motion refinement is based at least in part on a product of the signal-difference parameter array and the horizontal intermediate parameter array. ### Claim 9 (depends on 6) The video encoder of claim 6, further configured to perform: obtaining a first array of vertical gradients based on the first prediction signal array; obtaining a second array of vertical gradients based on a the second prediction signal array; and obtaining a vertical intermediate parameter array including performing a right bit shift on a sum of the first array of vertical gradients and the second array of vertical gradients; wherein the first-component motion refinement is a vertical motion refinement and is further based at least in part on the vertical intermediate parameter array. ### Claim 10 (depends on 9) The video encoder of claim 9, wherein the vertical motion refinement is based at least in part on a product of the signal-difference parameter array and the vertical intermediate parameter array. ### Claim 11 A video decoding method comprising: obtaining a first prediction signal array from a first reference picture and a second prediction signal array from a second reference picture; performing a right bit shift on the first prediction signal array to obtain a bit-shifted first prediction signal array and on the second prediction signal array to obtain a bit-shifted second prediction signal array; obtaining a signal-difference parameter array based on a difference between the bit-shifted first prediction signal array and the bit-shifted second prediction signal array; obtaining at least a first-component motion refinement based at least in part on the signal-difference parameter array; and generating a prediction of a current block in a video with bi-directional optical flow using at least the first-component motion refinement. ### Claim 12 (depends on 11) The method of claim 11, further comprising: obtaining a first array of horizontal gradients based on the first prediction signal array; obtaining a second array of horizontal gradients based on a the second prediction signal array; and obtaining a horizontal intermediate parameter array including performing a right bit shift on a sum of the first array of horizontal gradients and the second array of horizontal gradients; wherein the first-component motion refinement is a horizontal motion refinement and is further based at least in part on the horizontal intermediate parameter array. ### Claim 13 (depends on 12) The method of claim 12, wherein the horizontal motion refinement is based at least in part on a product of the signal-difference parameter array and the horizontal intermediate parameter array. ### Claim 14 (depends on 11) The method of claim 11, further comprising: obtaining a first array of vertical gradients based on the first prediction signal array; obtaining a second array of vertical gradients based on a the second prediction signal array; and obtaining a vertical intermediate parameter array including performing a right bit shift on a sum of the first array of vertical gradients and the second array of vertical gradients; wherein the first-component motion refinement is a vertical motion refinement and is further based at least in part on the vertical intermediate parameter array. ### Claim 15 (depends on 12) The method of claim 12, wherein obtaining the first array of horizontal gradients comprises padding samples outside the first prediction sample array with respective nearest boundary samples inside the first prediction sample array, and obtaining the second array of horizontal gradients comprises padding samples outside the second prediction sample array with respective nearest boundary samples inside the second prediction sample array. ### Claim 16 A video decoder comprising one or more processors configured to perform: obtaining a first prediction signal array from a first reference picture and a second prediction signal array from a second reference picture; performing a right bit shift on the first prediction signal array to obtain a bit-shifted first prediction signal array and on the second prediction signal array to obtain a bit-shifted second prediction signal array; obtaining a signal-difference parameter array based on a difference between the bit-shifted first prediction signal array and the bit-shifted second prediction signal array; obtaining at least a first-component motion refinement based at least in part on the signal-difference parameter array; and generating a prediction of a current block in a video with bi-directional optical flow using at least the first-component motion refinement. ### Claim 17 (depends on 16) The video decoder of claim 16, further configured to perform: obtaining a first array of horizontal gradients based on the first prediction signal array; obtaining a second array of horizontal gradients based on a the second prediction signal array; and obtaining a horizontal intermediate parameter array including performing a right bit shift on a sum of the first array of horizontal gradients and the second array of horizontal gradients; wherein the first-component motion refinement is a horizontal motion refinement and is further based at least in part on the horizontal intermediate parameter array. ### Claim 18 (depends on 17) The video decoder of claim 17, wherein the horizontal motion refinement is based at least in part on a product of the signal-difference parameter array and the horizontal intermediate parameter array. ### Claim 19 (depends on 16) The video decoder of claim 16, further configured to perform: obtaining a first array of vertical gradients based on the first prediction signal array; obtaining a second array of vertical gradients based on a the second prediction signal array; and obtaining a vertical intermediate parameter array including performing a right bit shift on a sum of the first array of vertical gradients and the second array of vertical gradients; wherein the first-component motion refinement is a vertical motion refinement and is further based at least in part on the vertical intermediate parameter array. ### Claim 20 (depends on 19) The video decoder of claim 19, wherein the vertical motion refinement is based at least in part on a product of the signal-difference parameter array and the vertical intermediate parameter array.