# US 12,604,010 B2 · Method, device, and medium for video processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,604,010 B2 - Document ID: US12604010B2 - Status: Granted - Kind: B2 - Application number: 18/607,278 - Filed: 2024-03-15 - Published: 2024-07-04 - Granted: 2026-04-14 - Earliest claimed: 2022-09-15 - Assignees: Beijing Bytedance Network Technology Co., Ltd.; Bytedance Inc. - Applicants: Beijing Bytedance Network Technology Co., Ltd.; Bytedance Inc. - Inventors: Zhipin Deng; Kai Zhang; Li Zhang - Web page: https://patentgrep.com/patent/US12604010B2 ## CPC - H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction - H04N 19/577: Motion compensation with bidirectional frame interpolation, i.e. using B-pictures - G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching} - H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction - H04N 19/136: Incoming video signal characteristics or properties - H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability - H04N 19/176: the region being a block, e.g. a macroblock - H04N 19/593: involving spatial prediction techniques - H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Embodiments of the present disclosure provide a solution for video processing. A method for video processing is proposed. The method comprises: determining, during a conversion between a target block of a video and a bitstream of the target block, a first prediction and a second prediction associated with the target block, the target block being applied with a multiple hypothesis prediction coding mode; applying a process associated with a template of the target block to the target block by blending the first and second predictions; and performing the conversion according to the process associated with the template. ## Claims ### Claim 1 A method of video processing, comprising: determining, during a conversion between a target block of a video and a bitstream of the video, a first prediction and a second prediction associated with the target block, the target block being applied with a multiple hypothesis prediction coding mode; applying a process associated with a template of the target block to the target block by blending the first and second predictions; and performing the conversion according to the process associated with the template, wherein the first prediction of the template is blended with the second prediction of the template, if one of the followings is calculated: a template matching cost of the target block, a template matching distortion of the target block, or a template matching error of the target block. ### Claim 2 (depends on 1) The method of claim 1, wherein the process associated with the template comprises at least one of: a generation of the template of the target block, a template matching cost of the target block, a template matching distortion of the target block, or a template matching error of the target block, or wherein the multiple hypothesis prediction coding mode comprises at least one of: a combined inter and intra prediction (CIIP) mode, a multiple hypothesis prediction (MHP) mode, or a geometric partitioning mode (GPM), or wherein at least one of the followings is used for a decode side method: a generation of the template of the target block, a template matching cost of the target block, a template matching distortion of the target block, or a template matching error of the target block; and wherein the decode side method comprises at least one of: a template matching, a bilateral matching, a candidate list reordering, a candidate index implicit derivation, or a weight index implicit derivation, or wherein the template is generated from a prior-coded sample other than a sample inside a current coding block of the target block. ### Claim 3 (depends on 1) The method of claim 1, wherein the second prediction is applied to the template, and wherein the method further comprising: constructing a final template by combining the first prediction and the second prediction of the template, or wherein an intra prediction of the template is applied to the template and combined with the first prediction or a reconstruction sample of the template, and wherein if the multiple hypothesis prediction coding mode combines an inter predication and an intra prediction together, the intra prediction is applied to the template, and wherein the first and second predictions of the template are combined for a template cost computation, or wherein an inter prediction of the template is applied to the template and combined with the first prediction or a reconstruction sample of the template, and wherein if the multiple hypothesis prediction coding mode combines a first inter prediction and a second inter prediction together, the inter prediction is applied to the template, and wherein the first and second predictions of the template are combined for a template cost computation. ### Claim 4 (depends on 1) The method of claim 1, wherein the number of prediction method applied to the template depends on the multiple hypothesis prediction coding mode, and wherein if the multiple hypothesis prediction coding mode blends from a first number of predictions, a second number of prediction method are blended for a template cost computation, or wherein a type of prediction applied to the template depends on the multiple hypothesis prediction coding mode, and wherein a first intra prediction mode used to generate a second intra prediction of the template is based on a second intra prediction mode of one of intra hypotheses of the target block, or wherein first motion information used to generate a second inter prediction of the template is based on second motion information of one of inter hypotheses of the target block. ### Claim 5 (depends on 1) The method of claim 1, wherein the first prediction of the template is generated based on a pre-defined rule, and wherein for bilateral matching, a first prediction of a first template is generated from samples in a first reference picture associated with the target block, and a first prediction of a second template is generated from samples in a second reference picture associated with the target block, or wherein for template matching, a first prediction of a first template is generated from a first set of neighbor samples of the target block in a current picture, and a first prediction of a second template is generated from a second set of neighbor samples in a reference picture, or wherein for GPM mode index reordering, the template is constructed by combining a first template and a second template, the first template is derived from a first prediction motion and the second template is derived from a second prediction motion. ### Claim 6 (depends on 1) The method of claim 1, further comprising: determining prediction data of the target block based on coding information of the target block; and performing the conversion comprises performing the conversion based on the prediction data, and wherein the prediction data comprises multiple hypothesis data of the target block, and the multiple hypothesis data of the target block is implicitly derived based on the coding information, and wherein a weight index or a weight value of a hypothesis is implicitly derived based on the coding information, and wherein the coding information comprises at least one of: mode information of a neighbor coded video unit associated with the target block, or a sample value of the neighbor coded video unit associated with the target block, or wherein the weight index for combining a first hypothesis and a second hypothesis for MHP is implicitly derived based on template matching cost, or wherein the hypothesis comprises at least one of: an additional hypothesis of the target block, or a base hypothesis of the target block, or wherein the multiple hypothesis data comprises at least one of: coding information of an additional hypothesis of the target block, or coding information of a base hypothesis of the target block, or wherein how to predict an additional hypothesis of the target block is implicitly derived, and the method further comprising: determining, based on the coding information, whether an additional hypothesis of the target block is coded based on advanced motion vector prediction (AMVP) or merge prediction. ### Claim 7 (depends on 6) The method of claim 6, wherein the determining is based on a decoder side template matching which is associated with a first template in a current picture and a second template in a reference picture, and wherein the first template in the current picture is constructed from a set of neighbor samples adjacent to the target block, or wherein a cost between the first template and the second template is used as a criterion to fulfill the determining, or wherein the determining is based on a decoder side bilateral matching which is associated with a first reference unit in a first reference picture and a second reference unit in a second reference picture, and wherein a cost between the first reference unit and the second reference unit is used as a criterion to fulfill the determining. ### Claim 8 (depends on 6) The method of claim 6, wherein the prediction data comprises AMVP prediction data of the target block, and the AMVP prediction data of the target block is implicitly derived based on the coding information, and wherein the AMVP prediction data comprises at least one of: a normal AMVP prediction, an AMVP based base hypothesis, or an AMVP based additional hypothesis, or wherein a predictor or a candidate index of an AMVP based prediction is implicitly derived based on the coding information, or wherein the predictor or the candidate index of an AMVP based hypothesis is implicitly derived based on the coding information, and the method further comprising: constructing a template for each AMVP predictor or AMVP candidate; for each AMVP predictor or AMVP candidate, determining a template matching cost between a first template of the target block and a second template of AMVP predictor or AMVP candidate; and determining a target AMVP predictor with minimum distortion as an implicitly derived AMVP predictor or AMVP candidate, or wherein a reference index of a hypothesis is implicitly derived based on the coding information, and the method further comprising: determining, based on a set of neighbor samples coded prior to the target block or a set of reference samples in a set of reference pictures, a distortion or cost from difference reference pictures; and determining the reference index with minimum cost. ### Claim 9 (depends on 6) The method of claim 6, wherein the prediction data comprises merge prediction data of the target block, and the merge prediction data of the target block is implicitly derived based on the coding information, and wherein the merge prediction data comprises at least one of: a normal merge prediction, a variant of the merge prediction, a merge based base hypothesis, or a merge based additional hypothesis, or wherein a predictor or a candidate index of a merge based prediction is implicitly derived based on the coding information, or wherein the predictor or the candidate index of a merge based hypothesis is implicitly derived based on the coding information, and the method further comprising: constructing a template for each merge predictor or merge candidate; for each merge predictor or merge candidate, determining a template matching cost between a first template of the target block and a second template of merge predictor or merge candidate; and determining a target merge predictor with minimum distortion as an implicitly derived merge predictor or merge candidate. ### Claim 10 (depends on 1) The method of claim 1, further comprising: determining motion data of a set of additional hypothesis candidates of the target block; reordering the motion data; and performing the conversion comprises performing the conversion based on the reordered motion data, and wherein the motion data comprises at least one of: motion information of advanced motion vector prediction (AMVP) coded additional hypothesis candidates, or motion information of merge coded additional hypothesis candidates, or wherein the motion data comprises at least one of: a motion vector, a prediction direction, a reference index, a bi-prediction with coding unit (CU)-level weight (BCW) index, half-pel interpolation filter information, or integer motion vector information, or wherein motion candidates of merge based additional hypotheses are reordered, and wherein the merge based additional hypotheses are generated based on a prediction list different from a regular merge list, or wherein the motion candidates to be reordered comprise at least one of: a merge candidate index, a motion vector, a BCW index, motion vector precision/resolution information, or half-pel interpolation filter information. ### Claim 11 (depends on 10) The method of claim 10, wherein motion candidates of AMVP based additional hypotheses are reordered, and wherein the AMVP based additional hypotheses are generated based on a prediction list different from a regular AMVP list, or wherein the motion candidates to be reordered comprise at least one of: a AMVP candidate index, a motion vector, a BCW index, IMV information, or half-pel interpolation filter information, or wherein the reordering is based on a rule of a template cost, and wherein the template cost is calculated from a set of neighbor samples of the target block in a current picture and a corresponding template in a reference picture, or wherein the reordering is based on a rule of a bilateral matching cost, and wherein the bilateral matching cost is calculated based on a corresponding block in a first reference picture and a second corresponding block in a second reference picture. ### Claim 12 (depends on 1) The method of claim 1, further comprising: determining an additional hypothesis of the target block; refining a set of motion candidates of the additional hypothesis of the target block by a decoder side method; and performing the conversion comprises performing the conversion based on the refined set of motion candidates. ### Claim 13 (depends on 12) The method of claim 12, wherein motion candidates of the additional hypothesis are firstly reordered, and then refined by the decoder side method, or wherein the set of motion candidates of the additional hypothesis are firstly refined by the decoder side method, then reordered, or wherein the set of motion candidates of the additional hypothesis are firstly refined by the decoder side method, without reordering, or wherein the set of motion candidates of the additional hypothesis are further refined by a template matching based method, or wherein the set of motion candidates of the additional hypothesis are further refined by a bilateral matching based method, or wherein whether and/or how to refine the set of motion candidates of the additional hypothesis depend on coding information of the target block, and wherein whether to refine the set of motion candidates of the additional hypothesis depends on a coding method of a base hypothesis of the target block, and wherein if the base hypothesis is template matching coded, the set of motion candidates of the additional hypothesis is further refined, or wherein if the base hypothesis is bilateral matching coded, the set of motion candidates of the additional hypothesis is further refined, or wherein regardless of whether the base hypothesis is bilateral matching coded or template matching coded, the set of motion candidates of the additional hypothesis is further refined. ### Claim 14 (depends on 12) The method of claim 12, wherein whether the set of motion candidates of the additional hypothesis is further refined or not is indicated through a syntax element and wherein the syntax element is further indicated indicating whether the set of motion candidates of the additional hypothesis is further refined by template matching or not, or wherein the syntax element is further indicated indicating whether the set of motion candidates of the additional hypothesis is further refined by bilateral matching or not, or wherein if the additional hypothesis is merge coded, the syntax element is further indicated indicating whether the set of motion candidates of the additional hypothesis if further refined by template matching or not, or wherein if the additional hypothesis is merge coded, the syntax element is further indicated indicating whether the set of motion candidates of the additional hypothesis if further refined by bilateral matching or not, or wherein if the additional hypothesis is AMVP coded, the syntax element is further indicated indicating whether the set of motion candidates of the additional hypothesis if further refined by template matching or not, or wherein if the additional hypothesis is AMVP coded, the syntax element is further indicated indicating whether the set of motion candidates of the additional hypothesis if further refined by bilateral matching or not. ### Claim 15 (depends on 1) The method of claim 1, further comprising: determining a prediction of a hypothesis of the target block based on virtual constructed motion data; and performing the conversion comprises performing the conversion based on the prediction, and wherein a prediction of an additional hypothesis of the target block is generated based on the virtual constructed motion data, or wherein the virtual constructed motion is based on an AMVP-MERGE candidate list, and wherein the AMVP-MERGE candidate list is generated based on at least one AMVP motion candidate information and at least one MERGE motion candidate information, and wherein the AMVP-MERGE candidate list is reordered, or wherein motion candidates in the AMVP-MERGE motion candidate list are refined, or wherein the virtual constructed motion data is based on a bi-directional virtual motion candidate, and wherein the bi-directional virtual motion candidate is generated based on a certain AMVP candidate list, or wherein the bi-directional virtual motion candidate is generated based on a certain MERGE candidate list, and wherein the certain MERGE candidate list comprises at least one of: a regular merge candidate list, a merge mode with motion vector difference (MMVD) candidate list, a template matching (TM) merge list, a GEO merge list, a combined inter intra prediction (CIIP) merge list, or a subblock merge list, or wherein the bi-directional virtual motion candidate is generated based on a reordered motion candidate list, or wherein the bi-directional virtual motion candidate is generated based on a reordered motion candidate list. ### Claim 16 (depends on 15) The method of claim 15, wherein the virtual constructed motion data is based on a uni-directional virtual motion candidate, and wherein the uni-directional virtual motion candidate is generated based on a certain AMVP candidate list, or wherein the uni-directional virtual motion candidate is generated based on a certain MERGE candidate list, and wherein the certain MERGE candidate list comprises at least one of: a regular merge candidate list, a MMVD candidate list, a TM merge list, a GEO merge list, a CIIP merge list, or a subblock merge list, or wherein the uni-directional virtual motion candidate is generated based on a reordered motion candidate list, or wherein the uni-directional virtual motion candidate is further refined, or wherein a prediction of an additional hypothesis of the target block is generated based on the virtual constructed motion data, and wherein if an AMVP-MERGE coded block is perceived as the base hypothesis, at least one additional hypothesis is applied to the AMVP-MERGE coded block, and a final prediction of the AMVP-MERGE coded block is generated by combining the base hypothesis and additional hypotheses together. ### Claim 17 (depends on 15) The method of claim 15, wherein at least one of: a bi-predicted additional hypothesis or a uni-predicted additional hypothesis is allowed for a multiple hypothesis prediction unit, and wherein a predetermined maximum number of bi-directional coded additional hypothesis is allowed for the multiple hypothesis prediction unit, and wherein the bi-directional hypothesis is generated based on a virtual generated prediction list, or wherein the bi-directional hypothesis is generated based on a bi-directional candidate from a certain merge list, or wherein the bi-directional hypothesis is generated based on a bi-directional candidate from a certain AMVP list, or wherein a predetermined maximum number of uni-directional coded additional hypotheses are allowed for the multiple hypothesis prediction unit, or wherein both uni-predicted additional hypothesis and bi-predicted additional hypothesis are allowed for the multiple hypothesis prediction unit, or wherein only bi-predicted additional hypothesis is allowed for the multiple hypothesis prediction unit, or wherein only uni-predicted additional hypothesis is allowed for the multiple hypothesis prediction unit. ### Claim 18 (depends on 1) The method of claim 1, wherein the conversion includes encoding the video unit into the bitstream, or wherein the conversion includes decoding the video unit from the bitstream. ### Claim 19 (depends on 1) The method of claim 1, further comprising: storing the bitstream in a non-transitory computer-readable recording medium. ### Claim 20 An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to perform a method comprising: determining, during a conversion between a target block of a video and a bitstream of the video, a first prediction and a second prediction associated with the target block, the target block being applied with a multiple hypothesis prediction coding mode; applying a process associated with a template of the target block to the target block by blending the first and second predictions; and performing the conversion according to the process associated with the template, wherein the first prediction of the template is blended with the second prediction of the template, if one of the followings is calculated: a template matching cost of the target block, a template matching distortion of the target block, or a template matching error of the target block. ### Claim 21 A non-transitory computer-readable storage medium storing instructions that cause a processor to perform a method comprising: determining, during a conversion between a target block of a video and a bitstream of the video, a first prediction and a second prediction associated with the target block, the target block being applied with a multiple hypothesis prediction coding mode; applying a process associated with a template of the target block to the target block by blending the first and second predictions; and performing the conversion according to the process associated with the template, wherein the first prediction of the template is blended with the second prediction of the template, if one of the followings is calculated: a template matching cost of the target block, a template matching distortion of the target block, or a template matching error of the target block.