# US 12,764,272 B2 · Fusion improvement and sub-pel precision mode for template matching related tools for video coding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,764,272 B2
- Document ID: US12764272B2
- Status: Granted
- Kind: B2
- Application number: 18/617,841
- Filed: 2024-03-27
- Published: 2024-10-03
- Granted: 2026-10-13
- Earliest claimed: 2023-03-28
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Po-Han Lin; Jian-Liang Lin; Vadim Seregin; Marta Karczewicz
- Web page: https://patentgrep.com/patent/US12764272B2

## CPC

- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/52: by predictive encoding
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N 19/80: Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation (H04N19/635, H04N19/86 take precedence)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method of encoding or decoding video data, the method comprising: applying a sub-pel precision mode to generate a prediction block for a current block of the video data, wherein a syntax element indicates that the sub-pel precision mode is applied to the current block and applying the sub-pel precision mode comprises: applying an interpolation filter to samples of a reference region to generate an array of samples at full-pel and sub-pel precision; and identifying, within the array, a reference template of the prediction block, wherein the reference template of the prediction block is a best match for a template of the current block within the array, wherein a template pattern defines a shape of the reference template of the prediction block and the template of the current block; and encoding or decoding the current block using the prediction block for the current block.

## Claims

### Claim 1

A method of encoding or decoding video data, the method comprising:
applying a sub-pel precision mode to generate a prediction block for a current block of the video data, wherein a syntax element indicates that the sub-pel precision mode is applied to the current block and applying the sub-pel precision mode comprises:
applying an interpolation filter to samples of a reference region to generate an array of samples at full-pel and sub-pel precision; and
identifying, within the array, a reference template of the prediction block, wherein the reference template of the prediction block is a best match for a template of the current block within the array, wherein a template pattern defines a shape of the reference template of the prediction block and the template of the current block;
generating a plurality of template matching (TM) candidates, wherein each of the TM candidates is associated with a respective prediction block and a respective motion vector predictor, and generating the plurality of TM candidates comprises, for each of the TM candidates:
applying the interpolation filter to samples of a reference region indicated by the motion vector predictor associated with the TM candidate to generate a respective array of samples at full-pel and sub-pel precision; and
identifying, within the respective array, a respective reference template of the respective prediction block, wherein the respective reference template of the respective prediction block is a best match for the template of the current block within the respective array;
generating the prediction block for the current block as a linear combination of corresponding samples in the prediction blocks associated with two or more TM candidates of the plurality of TM candidates, wherein generating the prediction block for the current block as the linear combination of the corresponding samples comprises:
for each of the two or more TM candidates, calculating a template cost for the TM candidate, and
generating the prediction block for the current block as a weighted average of the corresponding samples in the prediction blocks associated with the two or more TM candidates, comprising determining weights used in the weighted average based on the template cost for the TM candidates; and
encoding or decoding the current block using the prediction block for the current block.

### Claim 2 (depends on 1)

The method of claim 1, wherein the syntax element is signaled based on a non-fusion template matching mode being used for the current block.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
generating a list of template matching candidates, wherein each of the template matching candidates is associated with a different motion vector; and
selecting a template matching candidate from the list of template matching candidates,
wherein based on the sub-pel precision mode being used to encode or decode the current block, the list of template matching candidates is reduced relative to when the sub-pel precision mode is not used.

### Claim 4 (depends on 1)

The method of claim 1, wherein, based on the sub-pel precision mode being used for encoding or decoding the current block, a template matching fusion mode flag is not signaled.

### Claim 5 (depends on 1)

The method of claim 1, wherein the sub-pel precision is quarter pixel precision.

### Claim 6 (depends on 1)

The method of claim 1, wherein the weights are multiplicative inverses of template matching costs for the TM candidates.

### Claim 7 (depends on 1)

The method of claim 1, wherein a flag indicates whether the weights are derived based on the template cost or based on mean squared error (MSE) minimization.

### Claim 8 (depends on 1)

The method of claim 1, wherein the syntax element is a first syntax element and generating the prediction block for the current block as the linear combination of the corresponding samples comprises:
determining a weight derivation method from among a plurality of weight derivation methods;
applying the weight derivation method to derive weights of the two or more TM candidates; and
generating the prediction block for the current block as a weighted average of the corresponding samples in the prediction blocks associated with the two or more TM candidates using the weights of the two or more TM candidates,
wherein a second syntax element indicates the weight derivation method.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
each fusion mode of a plurality of available fusion modes is associated with a different combination of two or more TM candidates in a plurality of TM candidates, a bitstream indicates a subset of the available fusion modes, a quantity of fusion modes in the subset is M, a quantity of the available fusion modes is N, where M<N, the fusion modes in the subset of available fusion modes have minimum combined template costs among the available fusion modes,
applying the sub-pel precision mode comprises:
for each fusion mode of the subset of the available fusion modes:
for each TM candidate of the two or more TM candidates associated with the fusion mode:
applying the interpolation filter to samples of a reference region indicated by a motion vector predictor associated with the TM candidate to generate a respective array of samples at full-pel and sub-pel precision; and
identifying, within the respective array, a respective reference template of a respective prediction block for the TM candidate, wherein the respective reference template of the respective prediction block is a best match for the template of the current block within the respective array; and
generating, based on a combination of the respective prediction blocks for the two or more TM candidates associated with the fusion mode, a prediction block for the fusion mode; and
determining the prediction block for the current block from among the prediction blocks for the fusion modes.

### Claim 10 (depends on 1)

The method of claim 1, wherein each fusion mode of a plurality of fusion modes is associated with a different combination of two or more TM candidates in a plurality of TM candidates, applying the sub-pel precision mode comprises:
for each fusion mode of the plurality of fusion modes:
for each TM candidate of the two or more TM candidates associated with the fusion mode:
applying the interpolation filter to samples of a reference region indicated by a motion vector predictor associated with the TM candidate to generate a respective array of samples at full-pel and sub-pel precision; and
identifying, within the respective array, a respective reference template of a respective prediction block for the TM candidate, wherein the respective reference template of the respective prediction block for the TM candidate is a best match for the template of the current block within the respective array;
for each respective weighting set of a plurality of weighting sets:
generating a prediction block for the respective weighting set as a weighted average of corresponding samples in the prediction blocks of the two or more TM candidates associated with the fusion mode, wherein weights used in the weighted average are included in the respective weighting set; and
calculating a combined template cost of the prediction block for the respective weighting set; and
selecting a weighting set for the fusion mode from among the plurality of weighting sets based on a minimum of the combined template cost of the prediction blocks for the weighting sets; and
determining the prediction block for the current block from among the prediction blocks for the selected weighting sets for the plurality of fusion modes.

### Claim 11

A device for coding video data, the device comprising:
memory to store the video data; and
one or more processors implemented in circuitry, the one or more processors configured to:
apply a sub-pel precision mode to generate a prediction block for a current block of the video data, wherein a syntax element is signaled to indicate that the sub-pel precision mode is applied to the current block and the one or more processors are configured to, applying the sub-pel precision mode:
apply an interpolation filter to samples of a reference region to generate an array of samples at full-pel and sub-pel precision; and
identify, within the array, a reference template of the prediction block, wherein the reference template of the prediction block is a best match for a template of the current block within the array, wherein a template pattern defines a shape of the reference template of the prediction block and the template of the current block;
generate a plurality of template matching (TM) candidates, wherein each of the TM candidates is associated with a respective prediction block and a respective motion vector predictor, and the one or more processors are configured to, as part of generating the plurality of TM candidates, for each of the TM candidates:
apply the interpolation filter to samples of a reference region indicated by the motion vector predictor associated with the TM candidate to generate a respective array of samples at full-pel and sub-pel precision; and
identify, within the respective array, a respective reference template of the respective prediction block, wherein the respective reference template of the respective prediction block is a best match for the template of the current block within the respective array;
generate the prediction block for the current block as a linear combination of corresponding samples in the prediction blocks associated with two or more TM candidates of the plurality of TM candidates, wherein to generate the prediction block for the current block as the linear combination of the corresponding samples the processors are configured to:
for each of the two or more TM candidates, calculate a template cost for the TM candidate, and
generate the prediction block for the current block as a weighted average of the corresponding samples in the two or more TM candidates, wherein the processors are configured to determine the weights used in the weighted average based on the template cost for the TM candidate; and
encode or decode the current block using the prediction block for the current block.

### Claim 12 (depends on 11)

The device of claim 11, wherein the syntax element is signaled based on a non-fusion template matching mode being used for the current block.

### Claim 13 (depends on 11)

The device of claim 11, wherein the one or more processors are further configured to:
generate a list of template matching candidates, wherein each of the template matching candidates is associated with a different motion vector; and
select a template matching candidate from the list of template matching candidates,
wherein based on the sub-pel precision mode being used to encode or decode the current block, the list of template matching candidates is reduced relative to when the sub-pel precision mode is not used.

### Claim 14 (depends on 11)

The device of claim 11, wherein, based on the sub-pel precision mode being used for encoding or decoding the current block, a template matching fusion mode flag is not signaled.

### Claim 15 (depends on 11)

The device of claim 11, wherein the sub-pel precision is quarter pixel precision.

### Claim 16 (depends on 11)

The device of claim 11, wherein the weights are multiplicative inverses of template matching costs for the TM candidates.

### Claim 17 (depends on 11)

The device of claim 11, wherein a flag is signaled to indicate whether the weights are derived based on the template cost or based on mean squared error (MSE) minimization.

### Claim 18 (depends on 11)

The device of claim 11, wherein the syntax element is a first syntax element and the one or more processors are configured to, as part of generating the prediction block for the current block as the linear combination of the corresponding samples:
determine a weight derivation method from among a plurality of weight derivation methods;
apply the weight derivation method to derive weights of the two or more TM candidates; and
generate the prediction block for the current block as the weighted average of the corresponding samples in the prediction blocks associated with the two or more TM candidates using the weights of the two or more TM candidates,
wherein a second syntax element indicates the weight derivation method.

### Claim 19 (depends on 11)

The device of claim 11, wherein:
each fusion mode of a plurality of available fusion modes is associated with a different combination of two or more TM candidates in a plurality of TM candidates, a bitstream indicates a subset of the available fusion modes, a quantity of fusion modes in the subset is M, a quantity of the available fusion modes is N, where M<N, the fusion modes in the subset of available fusion modes have minimum combined template costs among the available fusion modes,
as part of applying the sub-pel precision mode, the one or more processors are configured to:
for each fusion mode of the subset of the available fusion modes:
for each TM candidate of the two or more TM candidates associated with the fusion mode:
apply the interpolation filter to samples of a reference region indicated by a motion vector predictor associated with the TM candidate to generate a respective array of samples at full-pel and sub-pel precision; and
identify, within the respective array, a respective reference template of a respective prediction block for the TM candidate, wherein the respective reference template of the respective prediction block is a best match for the template of the current block within the respective array; and
generate, based on a combination of the respective prediction blocks for the two or more TM candidates associated with the fusion mode, a prediction block for the fusion mode; and
determine the prediction block for the current block from among the prediction blocks for the fusion modes.

### Claim 20 (depends on 11)

The device of claim 11, wherein each fusion mode of a plurality of fusion modes is associated with a different combination of two or more TM candidates in a plurality of TM candidates, and the one or more processors are configured to, as part of applying the sub-pel precision mode:
for each fusion mode of the plurality of fusion modes:
for each TM candidate of the two or more TM candidates associated with the fusion mode:
apply the interpolation filter to samples of a reference region indicated by a motion vector predictor associated with the TM candidate to generate a respective array of samples at full-pel and sub-pel precision; and
identify, within the respective array, a respective reference template of a respective prediction block for the TM candidate, wherein the respective reference template of the respective prediction block is a best match for the template of the current block within the respective array;
for each respective weighting set of a plurality of weighting sets,
generate a prediction block for the respective weighting set as a weighted average of corresponding samples in the prediction blocks of the two or more TM candidates associated with the fusion mode, wherein weights used in the weighted average are included in the respective weighting set; and
calculate a combined template cost of the prediction block for the respective weighting set; and
select a weighting set for the fusion mode from among the plurality of weighting sets based on a minimum of the combined template cost of the prediction blocks for the weighting sets; and
determine the prediction block for the current block from among the prediction blocks for the selected weighting sets for the plurality of fusion modes.

### Claim 21 (depends on 11)

The device of claim 11, further comprising a display configured to display decoded video data.

### Claim 22 (depends on 11)

The device of claim 11, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

### Claim 23 (depends on 11)

The device of claim 11, wherein the device comprises a video decoder configured to decode the video data based on the prediction block.

### Claim 24 (depends on 11)

The device of claim 11, wherein the device comprises a video encoder configured to encode the video data based on the prediction block.