# US 12,003,736 B2 · Method and apparatus for estimating optical flow for motion compensation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,736 B2 - Document ID: US12003736B2 - Status: Granted - Kind: B2 - Application number: 17/673,922 - Filed: 2022-02-17 - Published: 2022-06-02 - Granted: 2024-06-04 - Earliest claimed: 2017-04-24 - Assignees: SK Telecom Co., Ltd. - Applicants: SK Telecom Co., Ltd. - Inventors: Jeong-yeon Lim; Hyo Song Kim; Se-hoon Son; Jae-seob Shin; Sun-young Lee - Web page: https://patentgrep.com/patent/US12003736B2 ## CPC - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/103: Selection of coding mode or of prediction mode - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/52: by predictive encoding - H04N 19/543: using regions - H04N 19/56: Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search - H04N 19/57: Motion estimation characterised by a search window with variable size or shape - H04N 19/573: Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction - H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures - H04N 19/182: the unit being a pixel - H04N 19/96: Tree coding, e.g. quad-tree coding - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Disclosed herein is a method for adaptive bidirectional optical flow estimation for inter prediction compensation during video encoding. The method aims to reduce complexity and/or cost of bidirectional optical flow (BIO) at a pixel level or a subblock level. ## Claims ### Claim 1 A method for encoding video data, comprising: determining a first motion vector indicating a first region corresponding to a current block in a first reference picture, and a second motion vector indicating a second region corresponding to the current block in a second reference picture; generating a prediction block for the current block by applying a bidirectional optical flow (BIO) process in a subblock basis; determining a residual block for the current block using the prediction block; and encoding the first motion vector, the second motion vector and the residual block for the current block in a bitstream, wherein the generating of the prediction block comprises: determining a BIO motion vector for each subblock constituting the current block; and generating a prediction value for each of pixels constituting a corresponding subblock based on the determined BIO motion vector, wherein the BIO motion vector is determined based on flow differences obtained for pixels within a square block surrounding the corresponding subblock, wherein the flow difference for a given pixel within the square block is calculated between a first point on the first reference picture and a second point on the second reference picture which correspond to the given pixel within the square block, wherein the prediction value for each of pixels is generated based on (1) an original prediction value derived by using the first motion vector and the second motion vector and (2) an offset value derived by using the BIO motion vector, wherein the BIO motion vector is determined as a vector that minimizes a weighted sum of squares of flow differences obtained for respective pixels within the square block surrounding the corresponding subblock, and wherein a higher weight is assigned to a flow difference obtained for a pixel located further inside the square block surrounding the corresponding subblock. ### Claim 2 (depends on 1) The method of claim 1, wherein the offset value is derived from a sum of a first value and a second value, the first value being a product between the difference of horizontal gradients for the first point and the second point and a horizontal component of the BIO motion vector, and the second value being a product between the difference of vertical gradients for the first point and the second point and a vertical component of the BIO motion vector. ### Claim 3 A method for decoding video data, comprising: determining a first motion vector indicating a first region corresponding to a current block in a first reference picture, and a second motion vector indicating a second region corresponding to the current block in a second reference picture; generating a prediction block for the current block by applying a bidirectional optical flow (BIO) process in a subblock basis; and reconstructing the current block using the prediction block, wherein the reconstructing of the current block includes: decoding a residual block for the current block from a bitstream; and adding the residual block to the prediction block to reconstruct the current block, and wherein the generating of the prediction block comprises: determining a BIO motion vector for each subblock constituting the current block; and generating a prediction value for each of pixels constituting a corresponding subblock based on the determined BIO motion vector, wherein the BIO motion vector is determined based on flow differences obtained for pixels within a square block surrounding the corresponding subblock, wherein the flow difference for a given pixel within the square block is calculated between a first point on the first reference picture and a second point on the second reference picture which correspond to the given pixel within the square block, wherein the prediction value for each of pixels is generated based on (1) an original prediction value derived by using the first motion vector and the second motion vector and (2) an offset value derived by using the BIO motion vector, wherein the BIO motion vector is determined as a vector that minimizes a weighted sum of squares of flow differences obtained for respective pixels within the square block surrounding the corresponding subblock, and wherein a higher weight is assigned to a flow difference obtained for a pixel located further inside the square block surrounding the corresponding subblock. ### Claim 4 (depends on 3) The method of claim 3, wherein the offset value is derived from a sum of a first value and a second value, the first value being a product between the difference of horizontal gradients for the first point and the second point and a horizontal component of the BIO motion vector, and the second value being a product between the difference of vertical gradients for the first point and the second point and a vertical component of the BIO motion vector. ### Claim 5 A non-transitory computer readable medium storing a bitstream containing encoded data for video data, the bitstream generated by processes of: determining a first motion vector indicating a first region corresponding to a current block in a first reference picture, and a second motion vector indicating a second region corresponding to the current block in a second reference picture; generating a prediction block for the current block by applying a bidirectional optical flow (BIO) process in a subblock basis; determining a residual block for the current block using the prediction block; and encoding the first motion vector, the second motion vector and the residual block for the current block in the bitstream, wherein the generating of the prediction block comprises: determining a BIO motion vector for each subblock constituting the current block; and generating a prediction value for each of pixels constituting a corresponding subblock based on the determined BIO motion vector, wherein the BIO motion vector is determined based on flow differences obtained for pixels within a square block surrounding the corresponding subblock, wherein the flow difference for a given pixel within the square block is calculated between a first point on the first reference picture and a second point on the second reference picture which correspond to the given pixel within the square block, wherein the prediction value for each of pixels is generated based on (1) an original prediction value derived by using the first motion vector and the second motion vector and (2) an offset value derived by using the BIO motion vector, wherein the BIO motion vector is determined as a vector that minimizes a weighted sum of squares of flow differences obtained for respective pixels within the square block surrounding the corresponding subblock, and wherein a higher weight is assigned to a flow difference obtained for a pixel located further inside the square block surrounding the corresponding subblock. ### Claim 6 (depends on 5) The non-transitory computer readable medium of claim 5, wherein the offset value is derived from a sum of a first value and a second value, the first value being a product between the difference of horizontal gradients for the first point and the second point and a horizontal component of the BIO motion vector, and the second value being a product between the difference of vertical gradients for the first point and the second point and a vertical component of the BIO motion vector.