# US 12,003,754 B2 · Error surface based sub-pixel accurate refinement method for decoder side motion vector refinement > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,003,754 B2 - Document ID: US12003754B2 - Status: Granted - Kind: B2 - Application number: 17/573,331 - Filed: 2022-01-11 - Published: 2022-04-28 - Granted: 2024-06-04 - Earliest claimed: 2018-07-02 - Assignees: Huawei Technologies Co., Ltd. - Applicants: Huawei Technologies Co., Ltd. - Inventors: Sriram Sethuraman; Jeeva Raj A; Sagar Kotecha - Web page: https://patentgrep.com/patent/US12003754B2 ## CPC - H04N 19/513: Processing of motion vectors - H04N 19/533: Motion estimation using multistep search, e.g. two-dimensional [2D]-log search or one-at-a-time search [OTS] - H04N 19/523: with sub-pixel accuracy - H04N 19/56: Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Given that decoder side motion vector refinement/derivation is a normative aspect of a coding system, the encoder will also have to perform the same error surface technique in order to not have any drift between the encoder's reconstruction and the decoder's reconstruction. Hence, all aspects of all embodiments are applicable to both encoding and decoding systems. In template matching, the refinement movement occurs only in the reference starting from the sub-pixel accurate center that is derived based on the explicitly signaled merge index or implicitly through cost evaluations. In bilateral matching (with or without averaged template), the refinements start in the reference lists L0 and L1 starting from the respective sub-pixel accurate centers that are derived based on the explicitly signaled merge index or implicitly through cost evaluations. ## Claims ### Claim 1 A decoder-side motion vector refinement method performed by an encoder for refining a motion vector for a block of a frame in a video, the method comprising: determining a target integer motion vector displacement by comparing integer distance costs associated with respective candidate integer motion vector displacements with respect to an initial motion vector for the block; determining a sub-pixel motion vector displacement for the motion vector based on the target integer motion vector displacement and the integer distance costs; determining a refined motion vector based on the target integer motion vector displacement, the sub-pixel motion vector displacement and the initial motion vector; wherein the sub-pixel motion vector displacement is determined as: x0=(E(−1,0)−E(1,0))/(2×(E(−1,0)+E(1,0)−2×E(0,0))), and y0=(E(0,−1)−E(0,1))/(2×(E(0,−1)+E(0,1)−2×E(0,0))), wherein: x0 and y0 are coordinates associated with the sub-pixel motion vector displacement relative to a center (0, 0), and E(−1, 0), E(1, 0), E(0, 0), E(0, −1) and E(0, 1) are the integer distance costs with respect to the initial motion vector and corresponding to candidate integer motion vector displacements represented by coordinates (−1, 0), (1, 0), (0, 0), (0, −1) and (0, 1), respectively, relative to the center (0,0); and the center (0,0) represents an integer motion vector displacement among the candidate integer motion vector displacements that has the lowest integer distance cost. ### Claim 2 (depends on 1) The method of claim 1, wherein the target integer motion vector displacement comprises a first motion vector displacement corresponding to a first reference picture list, and a second motion vector displacement corresponding to a second reference picture list, the method further comprising: determining the first motion vector displacement by comparing the integer distance costs associated with the respective candidate integer motion vector displacements corresponding to the first reference picture list; and determining the second motion vector displacement to be a negative of the first motion vector displacement. ### Claim 3 (depends on 1) The method of claim 1, wherein x0 is determined by performing at least one of a shift operation, a comparison operation, or an increment operation on E (−1, 0), E(1, 0) and E(0, 0). ### Claim 4 (depends on 1) The method of claim 1, wherein y0 is determined by performing at least one of a shift operation, a comparison operation, or an increment operation on E(0, −1), E(0, 1) and E(0, 0). ### Claim 5 (depends on 1) The method of claim 1, wherein the sub-pixel motion vector displacement is bounded between −0.5 and +0.5 pixel. ### Claim 6 (depends on 1) The method of claim 1, further comprising, prior to determining the sub-pixel motion vector displacement: determining whether the integer distance costs are available at a predetermined set of positions relative to the target integer motion vector displacement; and in the event that the integer distance costs are available at the predetermined set of positions relative to the target integer motion vector displacement, determining the sub-pixel motion vector displacement by performing calculation on the integer distance costs. ### Claim 7 (depends on 6) The method of claim 6, further comprising: in the event that the integer distance costs are not available at the predetermined set of positions relative to the target integer motion vector displacement, calculating the refined motion vector based on the target integer motion vector displacement and the initial motion vector. ### Claim 8 (depends on 1) The method of claim 1, wherein determining the target integer motion vector displacement comprising: calculating an integer distance cost for each candidate integer motion vector displacement in a plurality of candidate integer motion vector displacements; and selecting a candidate integer motion vector displacement corresponding to a lowest integer distance cost as the target integer motion vector displacement. ### Claim 9 An encoder apparatus comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, wherein the programming, when executed by the one or more processors, configures the apparatus to perform operations comprising: determining a target integer motion vector displacement by comparing integer distance costs associated with respective candidate integer motion vector displacements with respect to an initial motion vector for a block of a frame in a video; determining a sub-pixel motion vector displacement based on the target integer motion vector displacement and the integer distance costs; determining a refined motion vector based on the target integer motion vector displacement, the sub-pixel motion vector displacement and the initial motion vector; the sub-pixel motion vector displacement is determined as: x0=(E(−1,0)−E(1,0))/(2×(E(−1,0)+E(1,0)−2×E(0,0))), and y0=(E(0,−1)−E(0,1))/(2×(E(0,−1)+E(0,1)−2×E(0,0))), wherein: x0 and y0 are coordinates associated with the sub-pixel motion vector displacement relative to a center (0, 0), and E(−1, 0), E(1, 0), E(0, 0), E(0, −1) and E(0, 1) are the integer distance costs with respect to the initial motion vector and corresponding to candidate integer motion vector displacements represented by coordinates (−1, 0), (1, 0), (0, 0), (0, −1) and (0, 1), respectively, relative to the center (0,0); and the center (0,0) represents an integer motion vector displacement among the candidate integer motion vector displacements that has the lowest integer distance cost. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the target integer motion vector displacement comprises a first motion vector displacement corresponding to a first reference picture list, and a second motion vector displacement corresponding to a second reference picture list, and wherein the operations further comprise: determining the first motion vector displacement by comparing the integer distance costs associated with the respective candidate integer motion vector displacements corresponding to the first reference picture list; and determining the second motion vector displacement to be a negative of the first motion vector displacement. ### Claim 11 (depends on 9) The apparatus of claim 9, wherein x0 is determined by performing at least one of a shift operation, a comparison operation, or an increment operation on E (−1, 0), E(1, 0) and E(0, 0). ### Claim 12 (depends on 9) The apparatus of claim 9, wherein y0 is determined by performing at least one of a shift operation, a comparison operation, or an increment operation on E(0, −1), E(0, 1) and E(0, 0). ### Claim 13 (depends on 9) The apparatus of claim 9, wherein the sub-pixel motion vector displacement is bounded between −0.5 and +0.5 pixel. ### Claim 14 (depends on 9) The apparatus of claim 9, wherein the operations further comprise: prior to determining the sub-pixel motion vector displacement, determining whether the integer distance costs are available at a predetermined set of positions relative to the target integer motion vector displacement; and in the event that the integer distance costs are available at the predetermined set of positions relative to the target integer motion vector displacement, determining the sub-pixel motion vector displacement by performing calculation on the integer distance costs. ### Claim 15 (depends on 14) The apparatus of claim 14, wherein the operations further comprise: in the event that the integer distance costs are not available at the predetermined set of positions relative to the target integer motion vector displacement, calculating the refined motion vector based on the target integer motion vector displacement and the initial motion vector. ### Claim 16 (depends on 9) The apparatus of claim 9, wherein the one or more processors determine the target integer motion vector displacement by: calculating an integer distance cost for each candidate integer motion vector displacement in a plurality of candidate integer motion vector displacements; and selecting a candidate integer motion vector displacement corresponding to a lowest integer distance cost as the target integer motion vector displacement. ### Claim 17 A non-transitory computer-readable medium storing instructions which, when executed by one or more processors of an encoder, perform operations comprising: determining a target integer motion vector displacement by comparing integer distance costs associated with respective candidate integer motion vector displacements with respect to an initial motion vector for a block of a frame in a video; determining a sub-pixel motion vector displacement based on the target integer motion vector displacement and the integer distance costs; determining a refined motion vector based on the target integer motion vector displacement, the sub-pixel motion vector displacement and the initial motion vector; wherein the sub-pixel motion vector displacement is determined as: x0=(E(−1,0)−E(1,0))/(2×(E(−1,0)+E(1,0)−2×E(0,0))), and y0=(E(0,−1)−E(0,1))/(2×(E(0,−1)+E(0,1)−2×E(0,0))), wherein: x0 and y0 are coordinates associated with the sub-pixel motion vector displacement relative to a center (0, 0), and E(−1, 0), E(1, 0), E(0, 0), E(0, −1) and E(0, 1) are the integer distance costs with respect to the initial motion vector and corresponding to candidate integer motion vector displacements represented by coordinates (−1, 0), (1, 0), (0, 0), (0, −1) and (0, 1), respectively, relative to the center (0,0); and the center (0,0) represents an integer motion vector displacement among the candidate integer motion vector displacements that has the lowest integer distance cost. ### Claim 18 (depends on 17) The non-transitory computer-readable medium of claim 17, wherein the target integer motion vector displacement comprises a first motion vector displacement corresponding to a first reference picture list, and a second motion vector displacement corresponding to a second reference picture list, and wherein the operations further comprise: determining the first motion vector displacement by comparing the integer distance costs associated with the respective candidate integer motion vector displacements corresponding to the first reference picture list; and determining the second motion vector displacement to be a negative of the first motion vector displacement.