# US 12,114,009 B2 · Method and apparatus for adaptive reordering for reference frames > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,114,009 B2 - Document ID: US12114009B2 - Status: Granted - Kind: B2 - Application number: 17/932,333 - Filed: 2022-09-15 - Published: 2023-03-23 - Granted: 2024-10-08 - Earliest claimed: 2021-09-22 - Assignees: Tencent America LLC - Applicants: Tencent America LLC - Inventors: Liang Zhao; Xin Zhao; Shan Liu - Web page: https://patentgrep.com/patent/US12114009B2 ## CPC - H04N 19/573: Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/129: Scanning of coding units, e.g. zig-zag scan of transform coefficients or flexible macroblock ordering [FMO] - H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability - H04N 19/172: the region being a picture, frame or field - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/196: being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters (processing of motion vectors H04N19/513) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract This disclosure relates to adaptively reordering reference frames for video coding technologies. Template matching (TM) may be used to reorder the reference frames or reference frame pairs for each block by comparing the difference between the template of a current block and the template of a reference block with reference to the motion information of the spatial reference motion information (or spatial motion vectors) and/or temporal reference motion information (or temporal motion vectors). The difference between the template of current block and the template of the reference block may be calculated for each spatial reference motion information and/or temporal reference motion information, and marked as the score value of the associated reference frames or reference frame pairs. The available reference frames or reference frame pairs are ranked based on the score value. ## Claims ### Claim 1 A method for reordering reference frames for each block by template matching (TM), the method comprising: comparing a first template of a current block and a second template of a reference block for motion information, wherein the motion information comprises one or more of a spatial reference motion information and a temporal reference motion information; calculating a difference between the first template of the current block and the second template of the reference block; determining a value for an associated reference frame based in the calculated difference; and reordering the reference frames based on the determined value, wherein when multiple motion vectors are pointing to one of the reference frames, one of the motion vectors with a smallest difference is used for determining the value. ### Claim 2 (depends on 1) The method of claim 1, wherein the reference frames further comprise a reference frame pair. ### Claim 3 (depends on 1) The method of claim 1, wherein the TM comprises a decoder-side motion vector derivation for refining the motion information of the current block. ### Claim 4 (depends on 1) The method of claim 1, wherein the reordering further comprises: ranking available reference frames based on the value for each of the reference frames. ### Claim 5 (depends on 4) The method of claim 4, wherein when the values of multiple reference frames are equivalent, the ranking corresponds with a scan order of the spatial reference motion information or temporal reference motion information. ### Claim 6 (depends on 4) The method of claim 4, wherein when the values of multiple reference frames are equivalent, the ranking is based on a frequency of appearance of these reference frames used in the spatial reference motion information or temporal reference motion information. ### Claim 7 (depends on 1) The method of claim 1, wherein the calculating the difference comprises at least one of a sum of absolute differences (SAD), a sum of squared differences (SSD), a mean squared error (MSE), or a sum of absolute transform differences (SATD). ### Claim 8 (depends on 1) The method of claim 1, wherein each of the templates comprises a top or left neighboring block. ### Claim 9 (depends on 1) The method of claim 1, wherein the spatial reference motion information comprises one or more spatial motion vectors. ### Claim 10 (depends on 1) The method of claim 1, wherein the temporal reference motion information comprises one or more temporal motion vectors. ### Claim 11 (depends on 1) The method of claim 1, wherein: allowed unidirectional and bidirectional compound reference frames are ranked together by using the TM, and an index of the reference frames for the current block is signaled in a bitstream. ### Claim 12 (depends on 1) The method of claim 1, wherein; allowed single reference frames are ranked together by using the TM, and an index of the reference frame for the current block is signaled in a bitstream. ### Claim 13 An apparatus for encoding a video bitstream, the apparatus comprising: a memory storing instructions; and a processor in communication with the memory, wherein, when the processor executes the instructions, the processor is configured to cause the apparatus to: comparing a first template of a current block and a second template of a reference block for motion information, wherein the motion information comprises a spatial reference motion information or a temporal reference motion information; calculating a difference between the first template and the second template; determining a value for an associated reference frame based in the calculated difference; ranking available reference frames based on the value for each of the reference frames; and reordering the reference frames based on the ranking, wherein when multiple motion vectors are pointing to one of the reference frames, one of the motion vectors with a smallest difference is used for determining the value. ### Claim 14 (depends on 13) The apparatus according to claim 13, wherein when the values of multiple reference frames are equivalent, the ranking corresponds with a scan order of the spatial reference motion information or temporal reference motion information. ### Claim 15 (depends on 13) The apparatus according to claim 13, wherein when the values of multiple reference frames are equivalent, the ranking is based on a frequency of appearance of these reference frames used in the spatial reference motion information or temporal reference motion information. ### Claim 16 (depends on 13) The apparatus according to claim 13, wherein the calculating the difference comprises at least one of a sum of absolute differences (SAD), a sum of squared differences (SSD), a mean squared error (MSE), or a sum of absolute transform differences (SATD). ### Claim 17 A method of processing video data, the method comprising: performing, by a device comprising a memory storing instructions and a processor in communication with the memory, a conversion between video data and a bitstream by: comparing a first template of a current block and a second template of a reference block for motion information, wherein the motion information comprises a spatial reference motion information or a temporal reference motion information; calculating a difference between the first template and the second template; determining a value for an associated reference frame based in the calculated difference; ranking available reference frames based on the value for each of the reference frames; and reordering the reference frames based on the ranking, wherein when multiple motion vectors are pointing to one of the reference frames, one of the motion vectors with a smallest difference is used for determining the value. ### Claim 18 (depends on 17) The method according to claim 17, wherein when the values of multiple reference frames are equivalent, the ranking corresponds with a scan order of the spatial reference motion information or temporal reference motion information.