# US 12,192,445 B2 · Inter prediction method based on variable coefficient deep learning > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,445 B2 - Document ID: US12192445B2 - Status: Granted - Kind: B2 - Application number: 18/126,623 - Filed: 2023-03-27 - Published: 2023-07-27 - Granted: 2025-01-07 - Earliest claimed: 2020-09-28 - Assignees: Hyundai Motor Company; Kia Corporation; Ewha University-Industry Collaboration Foundation - Applicants: Hyundai Motor Company; Kia Corporation; Ewha University—Industry Collaboration Foundation - Inventors: Je Won Kang; Na Young Kim; Jung Kyung Lee; Seung Wook Park; Wha Pyeong Lim - Web page: https://patentgrep.com/patent/US12192445B2 ## CPC - H04N 19/109: among a plurality of temporal predictive coding modes - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G06T 9/00: Image coding (bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00) - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability - H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/513: Processing of motion vectors - H04N 19/52: by predictive encoding - H04N 19/54: using feature points or meshes - H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL ## Abstract An inter prediction method allows a variable coefficient deep learning model to adaptively learn characteristics of a video; transmits a variable coefficient deep learning model parameter generated from the learning from an image encoding device to an image decoding device; and refers to a virtual reference frame generated by the variable coefficient deep learning model. ## Claims ### Claim 1 A video decoding method performed by a video decoding apparatus, the video decoding method comprising: decoding variable coefficient values, an affine prediction flag, and an encoding mode from a bitstream, the affine prediction flag indicating whether affine motion prediction is applied to a current block, and the encoding mode being an encoding mode for motion information of the current block and indicating a merge mode or an advanced motion vector prediction (AMVP) mode; generating a virtual reference frame based on reference frames by using an interpolation model, wherein a variable coefficient network included in the interpolation model is set with the variable coefficient values; and generating a merge candidate of the current block based on the virtual reference frame and the reference frames when the encoding mode is the merge mode. ### Claim 2 (depends on 1) The video decoding method of claim 1, wherein the interpolation model is one of: a deep learning model configured to directly generate the virtual reference frame from the reference frames; a deep learning model configured to generate a mask from the reference frames and then perform weighed sum on the reference frames based on the mask to generate the virtual reference frame; or a deep learning model configured to generate an optical flow from the reference frames and then warp the reference frames based on the optical flow to generate the virtual reference frame. ### Claim 3 (depends on 1) The video decoding method of claim 1, wherein the variable coefficients values are transmitted from a video encoding apparatus and decoded by the video decoding apparatus, wherein the variable coefficients values are generated in advance by the video encoding apparatus updating the variable coefficient network of the interpolation model using a loss function based on a difference between an inferred frame generated by the interpolation model and an original frame. ### Claim 4 (depends on 1) The video decoding method of claim 1, wherein the interpolation model further includes a fixed coefficient network, and wherein fixed coefficient values of the fixed coefficient network are generated under pre-training based on an entire original reference frame and are set according to a predefined agreement. ### Claim 5 (depends on 4) The video decoding method of claim 4, wherein the variable coefficient network of the interpolation model is connected to a back end of the fixed coefficient network. ### Claim 6 (depends on 1) The video decoding method of claim 1, wherein the variable coefficient values are decoded per frame or at least one group of picture (GOP). ### Claim 7 (depends on 1) The video decoding method of claim 1, wherein the generating of the merge candidates includes: searching for spatial merge candidates; and searching for a temporal merge candidate in consideration of the virtual reference frame. ### Claim 8 (depends on 7) The video decoding method of claim 7, wherein the searching for the temporal merge candidate includes setting a zero motion vector as the merge candidate when both 1) a reference frame of the temporal merge candidate and 2) a picture in which there is a co-located block of the current block are the virtual reference frames. ### Claim 9 (depends on 1) The video decoding method of claim 1, further comprising: generating an affine AMVP candidate when the affine prediction flag is true and the encoding mode is the AMVP mode, wherein the generating of the affine AMVP candidate includes: adding an inherited affine AMVP candidate in consideration of the virtual reference frame; and adding a constructed affine AMVP candidate in consideration of the virtual reference frame. ### Claim 10 (depends on 9) The video decoding method of claim 9, wherein the adding of the inherited affine AMVP candidate includes adding a zero motion vector as the affine AMVP candidate when a reference picture of the inherited affine AMVP candidate and the reference picture of the current block are the same reference picture, and when the same reference picture is the virtual reference frame. ### Claim 11 (depends on 9) The video decoding method of claim 9, wherein the adding of the constructed affine AMVP candidate includes adding a zero motion vector as a reference picture when a reference picture of all control point motion vectors of the constructed affine AMVP candidate and the reference picture of the current block are the same reference picture, and when the same reference picture is the virtual reference frame. ### Claim 12 A video encoding method performed by a video encoding apparatus, the video encoding method comprising: acquiring previously generated variable coefficient values and a preset encoding mode, the encoding mode being an encoding mode for motion information of a current block and indicating a merge mode or an advanced motion vector prediction (AMVP) mode; generating a virtual reference frame from reference frames by using an interpolation model, wherein a variable coefficient network included in the interpolation model is set with the variable coefficient values; and generating a merge candidate of the current block based on the virtual reference frame and the reference frames when the encoding mode is the merge mode. ### Claim 13 (depends on 12) The video encoding method of claim 12, wherein the variable coefficient values are generated by updating the variable coefficient network of the interpolation model, and wherein updating of the variable coefficient network uses a loss function based on a difference between an inferred frame generated by the interpolation model and an original frame. ### Claim 14 (depends on 12) The video encoding method of claim 12, wherein the interpolation model further includes a fixed coefficient network, and fixed coefficient values of the fixed coefficient network are generated under pre-training based on an entire original reference frame and are set according to a predefined agreement. ### Claim 15 (depends on 12) The video encoding method of claim 12, wherein the variable coefficient values are encoded per frame or at least one group of picture (GOP). ### Claim 16 A method for providing video data to a video decoding device, the method comprising: encoding the video data into a bitstream; and transmitting the bitstream to the video decoding device, wherein encoding the video data comprises: acquiring previously generated variable coefficient values and a preset encoding mode from a bitstream, the encoding mode being an encoding mode for motion information of a current block and indicating a merge mode or an advanced motion vector prediction (AMVP) mode; generating a virtual reference frame from reference frames using an interpolation model, wherein a variable coefficient network included in the interpolation model is set with the variable coefficient values; and generating a merge candidate of the current block based on the virtual reference frame and the reference frames when the encoding mode is the merge mode.