# US 12,113,987 B2 · Multi-pass decoder-side motion vector refinement > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,113,987 B2 - Document ID: US12113987B2 - Status: Granted - Kind: B2 - Application number: 17/556,142 - Filed: 2021-12-20 - Published: 2022-06-23 - Granted: 2024-10-08 - Earliest claimed: 2020-12-22 - Assignees: QUALCOMM Incorporated - Applicants: QUALCOMM Incorporated - Inventors: Zhi Zhang; Han Huang; Chun-Chi Chen; Yan Zhang; Vadim Seregin; Marta Karczewicz - Web page: https://patentgrep.com/patent/US12113987B2 ## CPC - H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability - H04N 19/192: the adaptation method, adaptation tool or adaptation type being iterative or recursive - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking - H04N 19/136: Incoming video signal characteristics or properties - H04N 19/157: Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/186: the unit being a colour or a chrominance component - H04N 19/513: Processing of motion vectors - H04N 19/521: {for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors} - H04N 19/96: Tree coding, e.g. quad-tree coding - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract An example device includes memory configured to store video data and one or more processors coupled to the memory. The one or more processors are configured to apply a multi-pass DMVR to a motion vector for a block of the video data to determine at least one refined motion vector and decode the block based on the at least one refined motion vector. The multi-pass DMVR includes a block-based first pass, a sub-block-based second pass, and a sub-block-based third pass. ## Claims ### Claim 1 A method of decoding video data, the method comprising: applying a multi-pass decoder-side motion vector refinement (DMVR) to a motion vector for a block of the video data to determine at least one refined motion vector; and decoding the block based on the at least one refined motion vector, wherein the multi-pass DMVR comprises: a first pass, the first pass being block-based and applied to the block of the video data, wherein applying the first pass comprises applying bi-lateral matching to the block and deriving at least one first refined motion vector for the block of the video data; a second pass, the second pass being sub-block-based and applied to at least one second pass sub-block of the block of the video data, wherein a second pass sub-block width is equal to or less than a width of the block of the video data, and a second pass sub-block height is equal to or less than a height of the block of the video data, wherein the second pass uses the at least one first refined motion vector as an initial motion vector for the second pass, and wherein applying the second pass comprises applying bi-lateral matching to the at least one second pass sub-block and deriving at least one second refined motion vector for at least one respective second pass sub-block; and a third pass, the third pass being sub-block-based and applied to at least one third pass sub-block of the block of the video data, the third pass sub-block being a sub-block of the at least one respective second pass sub-block, wherein a third pass sub-block width is equal to or less than the second pass sub-block width, and a third pass sub-block height is equal to or less than the second pass sub-block height, and wherein the third pass uses the at least one second refined motion vector as an initial motion vector for the at least one third pass sub-block, wherein applying the third pass comprises applying bi-directional optical flow (BDOF) to the at least one third pass sub-block and deriving at least one third refined motion vector for at least one respective third pass sub-block, and wherein the at least one refined motion vector is based on the at least one third refined motion vector. ### Claim 2 (depends on 1) The method of claim 1, wherein the at least one second pass sub-block has a predetermined maximum width of 16 luma samples and a predetermined maximum height of 16 luma samples. ### Claim 3 (depends on 1) The method of claim 1, wherein the at least one third pass sub-block has a predetermined maximum width of 8 luma samples and a predetermined maximum height of 8 luma samples. ### Claim 4 (depends on 1) The method of claim 1, wherein a delta motion value range for at least one of the first pass or the second pass is [−8, 8] in a horizontal direction and [−8, 8] in a vertical direction and a delta motion value range for the third pass is [−2, 2] in the horizontal direction and [−2, 2] in the vertical direction. ### Claim 5 (depends on 1) The method of claim 1, wherein the block of the video data is a first block, the method further comprising applying a shortened multi-pass DMVR to a motion vector for a second block of the video data, comprising: determining to skip a given pass of the multi-pass DMVR for the second block; and skipping the given pass of the multi-pass DMVR for the second block based on the determining to skip the given pass of the multi-pass DMVR for the second block. ### Claim 6 (depends on 5) The method of claim 5, wherein the determining to skip the given pass is based on a result of a preceding pass. ### Claim 7 (depends on 1) The method of claim 1, wherein the block of the video data is a first block, the method further comprising applying a shortened multi-pass DMVR to a motion vector for a second block of the video data, comprising: applying a given sub-block-based pass of the multi-pass DMVR to a first sub-area of the second block of the video data, the first sub-area comprising a first one or more sub-blocks of the second block and deriving at least one refined motion vector for the first sub-area from applying the given sub-block-based pass of the multi-pass DMVR to the first sub-area; determining to skip the given sub-block-based pass of the multi-pass DMVR for a second sub-area of the second block of the video data, the second sub-area comprising a second one or more sub-blocks of the second block, each of the first one or more sub-blocks of the second block being different than each of the second one or more sub-blocks of the second block; and skipping the given sub-block-based pass of the multi-pass DMVR for the second sub-area of the second block based on the determining to skip the given sub-block-based pass of the multi-pass DMVR for the second sub-area of the second block, and deriving at least one refined motion vector for the second sub-area based on a refined motion vector from a preceding pass of the multi-pass DMVR to the second sub-area. ### Claim 8 (depends on 7) The method of claim 7, wherein the determining to skip the given sub-block-based pass is based on a result of the preceding pass. ### Claim 9 (depends on 1) The method of claim 1, wherein the block is a first block of the video data, the method further comprising: determining not to apply DMVR to a second block of the video data; based on the determining to not apply DMVR to the second block, skipping all passes of the multi-pass DMVR for the second block; and decoding the second block based on an initial motion vector for the second block. ### Claim 10 A device for decoding video data, the device comprising: memory configured to store the video data; and one or more processors implemented in circuitry and communicatively coupled to the memory, the one or more processors being configured to: apply a multi-pass decoder-side motion vector refinement (DMVR) to a motion vector for a block of the video data to determine at least one refined motion vector; and decode the block based on the at least one refined motion vector, wherein the multi-pass DMVR comprises: a first pass, the first pass being block-based and applied to the block of the video data, wherein applying the first pass comprises applying bi-lateral matching to the block and deriving at least one first refined motion vector for the block of the video data; a second pass, the second pass being sub-block-based and applied to at least one second pass sub-block of the block of the video data, wherein a second pass sub-block width is equal to or less than a width of the block of the video data, and a second pass sub-block height is equal to or less than a height of the block of the video data, wherein the second pass uses the at least one first refined motion vector as an initial motion vector for the second pass, and wherein applying the second pass comprises applying bi-lateral matching to the at least one second pass sub-block and deriving at least one second refined motion vector for at least one respective second pass sub-block; and a third pass, the third pass being sub-block-based and applied to at least one third pass sub-block of the block of the video data, the third pass sub-block being a sub-block of the at least one respective second pass sub-block, wherein a third pass sub-block width is equal to or less than the second pass sub-block width, and a third pass sub-block height is equal to or less than the second pass sub-block height, and wherein the third pass uses the at least one second refined motion vector as an initial motion vector for the at least one third pass sub-block, wherein applying the third pass comprises applying bi-directional optical flow (BDOF) to the at least one third pass sub-block and deriving at least one third refined motion vector for at least one respective third pass sub-block, and wherein the at least one refined motion vector is based on the at least one third refined motion vector. ### Claim 11 (depends on 10) The device of claim 10, wherein the at least one second pass sub-block has a predetermined maximum width of 16 luma samples and a predetermined maximum height of 16 luma samples. ### Claim 12 (depends on 10) The device of claim 10, wherein the at least one third pass sub-block has a predetermined maximum width of 8 luma samples and a predetermined maximum height of 8 luma samples. ### Claim 13 (depends on 10) The device of claim 10, wherein a delta motion value range for at least one of the first pass or the second pass is [−8, 8] in a horizontal direction and [−8, 8] in a vertical direction and a delta motion value range for the third pass is [−2, 2] in the horizontal direction and [−2, 2] in the vertical direction. ### Claim 14 (depends on 10) The device of claim 10, wherein the block of the video data is a first block, wherein the one or more processors are configured to apply a shortened multi-pass DMVR to a motion vector for a second block of the video data, and wherein to apply the shortened multi-pass DMVR to the motion vector for the second block, the one or more processors are configured to: determine to skip a given pass of the multi-pass DMVR for the second block; and skip the given pass of the multi-pass DMVR for the second block based on the determination to skip the given pass of the multi-pass DMVR for the second block. ### Claim 15 (depends on 14) The device of claim 14, wherein the one or more processors are configured to determine to skip the given pass is based on a result of a preceding pass. ### Claim 16 (depends on 10) The device of claim 10, wherein the block of the video data is a first block, wherein the one or more processors are configured to apply a shortened multi-pass DMVR to a motion vector for a second block of the video data, and wherein to apply the shortened multi-pass DMVR to the motion vector for the second block, the one or more processors are configured to: apply a given sub-block-based pass of the multi-pass DMVR for a first sub-area of the second block of the video data, the first sub-area comprising a first one or more sub-blocks of the second block and derive at least one refined motion vector for the first sub-area from applying the given sub-block-based pass of the multi-pass DMVR to the first sub-area; determine to skip the given sub-block-based pass of the multi-pass DMVR for a second sub-area of the second block of the video data, the second sub-area comprising a second one or more sub-blocks of the second block, each of the first one or more sub-blocks of the second block being different than each of the second one or more sub-blocks of the second block; and skip the given sub-block-based pass of the multi-pass DMVR for the second sub-area of the second block based on the determination to skip the given sub-block-based pass of the multi-pass DMVR for the second sub-area of the second block and derive at least one refined motion vector for the second sub-area based on a refined motion vector from a preceding pass of the multi-pass DMVR to the second sub-area. ### Claim 17 (depends on 16) The device of claim 16, wherein the one or more processors are configured to determine to skip the given sub-block-based pass is based on a result of the preceding pass. ### Claim 18 (depends on 10) The device of claim 10, wherein the block is a first block of the video data, the one or more processors are further configured to: determine not to apply DMVR to a second block of the video data; based on the determination to not apply DMVR to the second block, skip all passes of the multi-pass DMVR for the second block; and decode the second block based on an initial motion vector for the second block. ### Claim 19 A non-transitory computer-readable storage medium storing instructions, which, when executed, cause one or more processors to: apply a multi-pass decoder-side motion vector refinement (DMVR) to a motion vector for a block of video data to determine at least one refined motion vector; and decode the block based on the at least one refined motion vector, wherein the multi-pass DMVR comprises: a first pass, the first pass being block-based and applied to the block of the video data, wherein applying the first pass comprises applying bi-lateral matching to the block and deriving at least one first refined motion vector for the block of the video data; a second pass, the second pass being sub-block-based and applied to at least one second pass sub-block of the block of the video data, wherein a second pass sub-block width is equal to or less than a width of the block of the video data, and a second pass sub-block height is equal to or less than a height of the block of the video data, wherein the second pass uses the at least one first refined motion vector as an initial motion vector for the second pass, and wherein applying the second pass comprises applying bi-lateral matching to the at least one second pass sub-block and deriving at least one second refined motion vector for at least one respective second pass sub-block; and a third pass, the third pass being sub-block-based and applied to at least one third pass sub-block of the block of the video data, the third pass sub-block being a sub-block of the at least one respective second pass sub-block, wherein a third pass sub-block width is equal to or less than the second pass sub-block width, and a third pass sub-block height is equal to or less than the second pass sub-block height, and wherein the third pass uses the at least one second refined motion vector as an initial motion vector for the at least one third pass sub-block, wherein applying the third pass comprises applying bi-directional optical flow (BDOF) to the at least one third pass sub-block and deriving at least one third refined motion vector for at least one respective third pass sub-block, and wherein the at least one refined motion vector is based on the at least one third refined motion vector. ### Claim 20 A device for coding video data, the device comprising: means for applying a multi-pass decoder-side motion vector refinement (DMVR) to a motion vector for a block of the video data to determine at least one refined motion vector; and means for decoding the block based on the at least one refined motion vector, wherein the multi-pass DMVR comprises: a first pass, the first pass being block-based and applied to the block of the video data, wherein applying the first pass comprises applying bi-lateral matching to the block and deriving at least one first refined motion vector for the block of the video data; a second pass, the second pass being sub-block-based and applied to at least one second pass sub-block of the block of the video data, wherein a second pass sub-block width is equal to or less than a width of the block of the video data, and a second pass sub-block height is equal to or less than a height of the block of the video data, wherein the second pass uses the at least one first refined motion vector as an initial motion vector for the second pass, and wherein applying the second pass comprises applying bi-lateral matching to the at least one second pass sub-block and deriving at least one second refined motion vector for at least one respective second pass sub-block; and a third pass, the third pass being sub-block-based and applied to at least one third pass sub-block of the block of the video data, the third pass sub-block being a sub-block of the at least one respective second pass sub-block, wherein a third pass sub-block width is equal to or less than the second pass sub-block width, and a third pass sub-block height is equal to or less than the second pass sub-block height, and wherein the third pass uses the at least one second refined motion vector as an initial motion vector for the at least one third pass sub-block, wherein applying the third pass comprises applying bi-directional optical flow (BDOF) to the at least one third pass sub-block and deriving at least one third refined motion vector for at least one respective third pass sub-block, and wherein the at least one refined motion vector is based on the at least one third refined motion vector.