# US 2025/0097454 A1 · Template-matching based motion shift refinement for subblock based motion vector (mv) predictor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0097454 A1
- Document ID: US20250097454A1
- Status: Published application
- Kind: A1
- Application number: 18/962,820
- Filed: 2024-11-27
- Published: 2025-03-20
- Earliest claimed: 2022-07-11
- Assignees: Tencent America LLC
- Applicants: Tencent America LLC
- Inventors: Lien-Fei CHEN; Guichun LI; Xin ZHAO; Shan LIU
- Web page: https://patentgrep.com/patent/US20250097454A1

## CPC

- H04N 19/52: by predictive encoding
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/109: among a plurality of temporal predictive coding modes
- H04N 19/139: Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
- H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
- H04N 19/167: Position within a video image, e.g. region of interest [ROI]
- H04N 19/172: the region being a picture, frame or field
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/43: Hardware specially adapted for motion estimation or compensation
- H04N 19/88: involving rearrangement of data among different coding units, e.g. shuffling, interleaving, scrambling or permutation of pixel data or permutation of transform coefficient data among different blocks
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Aspects of the disclosure provide a method and an apparatus including processing circuitry that decodes a current block in a current picture with a subblock-based temporal motion vector prediction (SbTMVP) mode. A first collocated block in a first collocated picture is determined based on a first displacement vector candidate of the current block corresponding to a first SbTMVP candidate. The processing circuitry determines first motion information of a current template of the current block based on one or more pieces of motion information of the first collocated block or a neighboring block of the first collocated block. The processing circuitry determines one of a first reference template and a first subblock reference template in a first reference picture based on the first motion information and determines a first template matching cost based on the current template and the one of the first reference template and the first subblock reference template.

## Claims

### Claim 1

(canceled)

### Claim 2

A method of decoding, comprising:
decoding prediction information of a current block in a current picture from a coded bitstream, the prediction information indicating a subblock-based temporal motion vector prediction (SbTMVP) mode; and
performing template matching (TM) by
determining a first collocated block in a first collocated picture based on a first motion shift candidate of the current block;
determining first motion information of a current template of the current block based on one or more pieces of motion information of the first collocated block, the first motion information of the current template indicating one of a first reference template and a first subblock reference template in a first reference picture relative to the current template that is in the current picture;
determining the one of the first reference template and the first subblock reference template associated with the first motion shift candidate based on the first motion information of the current template; and
determining a first TM cost based on the current template and the one of the first reference template and the first subblock reference template.

### Claim 3 (depends on 2)

The method of claim 2, wherein
motion shift candidates of the current block include the first motion shift candidate; and
the performing the TM includes performing the TM by
for each of the motion shift candidates that is different from the first motion shift candidate,
determining a collocated block in one of at least one collocated picture based on the respective motion shift candidate of the current block, the at least one collocated picture including the first collocated picture;
determining motion information of the current template based on one or more pieces of motion information of the collocated block associated with the respective motion shift candidate;
determining one of a reference template and a subblock reference template in a reference picture based on the motion information of the current template; and
determining a TM cost based on the current template and the one of the reference template and the subblock reference template associated with the respective motion shift candidate; and
determining a motion shift candidate to be used in the SbTMVP mode based on the TM costs that include the first TM cost; and
the method further includes:
determining an SbTMVP candidate based on the determined motion shift candidate, a respective one of the motion shift candidates of the current block corresponding to each of SbTMVP candidates of the current block; and
reconstructing the current block based on the determined SbTMVP candidate.

### Claim 4 (depends on 2)

The method of claim 2, wherein
the one of the first reference template and the first subblock reference template is the first reference template;
the determining the first motion information of the current template includes determining the first motion information of the current template as motion information of a center position of the first collocated block; and
the determining the first reference template includes determining the first reference template associated with the first motion shift candidate based on the motion information of the center position of the first collocated block.

### Claim 5 (depends on 2)

The method of claim 2, wherein
the one of the first reference template and the first subblock reference template is the first subblock reference template;
the current template includes a first subblock template and a second subblock template;
the first motion information of the current template includes a first piece of motion information of the first subblock template and a second piece of motion information of the second subblock template that is different from the first piece of motion information;
the determining the first motion information of the current template includes determining the first piece of motion information of the first subblock template from a first subblock and determining the second piece of motion information of the second subblock template from a second subblock, the first subblock and the second subblock being different subblocks in the first collocated block;
the method further includes:
determining the first subblock reference template in the first reference picture based on the first piece of motion information of the first subblock template;
determining a second subblock reference template based on the second piece of motion information of the second subblock template, the second subblock reference template being in one of the first reference picture and in a second reference picture; and
determining a predictor template based on the first subblock reference template in the first reference picture and the second subblock reference template; and
the determining the first TM cost includes determining the first TM cost based on the current template and the predictor template.

### Claim 6 (depends on 5)

The method of claim 5, wherein the determining the first piece of motion information of the first subblock template comprises:
determining the first piece of motion information of the first subblock template based on motion information of the first subblock in the first collocated block, the first subblock in the first collocated block being collocated with a subblock in the current block that is adjacent to the first subblock template.

### Claim 7 (depends on 5)

The method of claim 5, wherein
the first piece of motion information of the first subblock template includes a first motion vector (MV) pointing to the first subblock reference template in the first reference picture and a second MV pointing to a first subblock reference template in the second reference picture; and
the determining the first subblock reference template in the first reference picture includes determining the first subblock reference template in the first reference picture based on the first MV;
the method includes determining the first subblock reference template in the second reference picture based on the second MV;
the determining the predictor template includes determining a first predictor subblock template using bi-prediction that is based on the first subblock reference template in the first reference picture and the first subblock reference template in the second reference picture.

### Claim 8 (depends on 5)

The method of claim 5, wherein
the second subblock template is predicted using uni-prediction;
the second piece of motion information of the second subblock template includes a MV pointing to the second subblock reference template in the first reference picture; and
the determining the predictor template includes determining a second predictor subblock template using uni-prediction that is based on the second subblock reference template in the first reference picture.

### Claim 9 (depends on 5)

The method of claim 5, wherein
the first subblock template is predicted using bi-prediction and the second subblock template is predicted using uni-prediction;
the first piece of motion information of the first subblock template includes a first motion vector (MV) pointing to the first subblock reference template in the first reference picture and a second MV pointing to a first subblock reference template in a second reference picture;
the second piece of motion information of the second subblock template includes an MV pointing to the second subblock reference template in the first reference picture; and
the determining the predictor template includes:
determining a first predictor subblock template using the bi-prediction that is based on the first subblock reference template in the first reference picture and the first subblock reference template in the second reference picture; and
determining a second predictor subblock template using the uni-prediction that is based on the second subblock reference template in the first reference picture.

### Claim 10 (depends on 9)

The method of claim 9, wherein the determining the predictor template comprises:
determining the predictor template based on the first predictor subblock template and the second predictor subblock template.

### Claim 11 (depends on 9)

The method of claim 9, wherein the determining the first TM cost comprises:
determining a first sub-TM cost based on the first subblock template and the first predictor subblock template;
determining a second sub-TM cost based on the second subblock template and the second predictor subblock template; and
determining the first TM cost based on the first sub-TM cost and the second sub-TM cost.

### Claim 12 (depends on 5)

The method of claim 5, wherein the determining the first motion information of the current template of the first subblock template comprises:
when the first subblock template is above the current block, determining the first piece of motion information of the first subblock template based on motion information of subblocks in a top row in the first collocated block; and
when the first subblock template is a left neighbor of the current block, determining the first piece of motion information of the first subblock template based on motion information of subblocks in a left most column in the first collocated block.

### Claim 13 (depends on 3)

The method of claim 3, wherein the determining the motion shift candidate comprises:
reordering the motion shift candidates based on the corresponding TM costs; and
selecting the motion shift candidate from the reordered motion shift candidates.

### Claim 14 (depends on 13)

The method of claim 13, wherein the selecting the motion shift candidate comprises:
selecting the motion shift candidate as one of the reordered motion shift candidates that corresponds to the smallest TM cost in the TM costs.

### Claim 15 (depends on 3)

The method of claim 3, wherein
the at least one collocated picture includes a plurality of collocated pictures; and
a reference picture of each of the motion shift candidates is one of the plurality of collocated pictures.

### Claim 16 (depends on 15)

The method of claim 15, further comprising:
determining the motion shift candidates based on a set of candidates including at least one of (i) a motion vector (MV) of a spatial neighboring block of the current block, (ii) an MV of a spatial neighboring subblock of the current block, (iii) a history-based MV predictor (HMVP), or (iv) a candidate in a merge candidate list that includes at least one of (a) a spatial candidate, (b) a temporal candidate, (c) a HMVP candidate, (d) a pairwise average candidate, or (e) a zero MV.

### Claim 17

A method of encoding, comprising:
performing template matching (TM) by
determining a first collocated block in a first collocated picture based on a first motion shift candidate of a current block in a current picture, the current block being encoded using a subblock-based temporal motion vector prediction (SbTMVP) mode;
determining first motion information of a current template of the current block based on one or more pieces of motion information of the first collocated block, the first motion information of the current template indicating one of a first reference template and a first subblock reference template in a first reference picture relative to the current template that is in the current picture;
determining the one of the first reference template and the first subblock reference template associated with the first motion shift candidate based on the first motion information of the current template; and
determining a first TM cost based on the current template and the one of the first reference template and the first subblock reference template.

### Claim 18 (depends on 17)

The method of claim 17, wherein
motion shift candidates of the current block include the first motion shift candidate; and
the performing the TM includes performing the TM by
for each of the motion shift candidates that is different from the first motion shift candidate,
determining a collocated block in one of at least one collocated picture based on the respective motion shift candidate of the current block, the at least one collocated picture including the first collocated picture;
determining motion information of the current template based on one or more pieces of motion information of the collocated block associated with the respective motion shift candidate;
determining one of a reference template and a subblock reference template in a reference picture based on the motion information of the current template; and
determining a TM cost based on the current template and the one of the reference template and the subblock reference template associated with the respective motion shift candidate; and
determining a motion shift candidate to be used in the SbTMVP mode based on the TM costs that include the first TM cost; and
the method further includes:
determining an SbTMVP candidate based on the determined motion shift candidate, a respective one of the motion shift candidates of the current block corresponding to each of SbTMVP candidates of the current block; and
encoding the current block based on the determined SbTMVP candidate.

### Claim 19 (depends on 17)

The method of claim 17, wherein
the one of the first reference template and the first subblock reference template is the first reference template;
the determining the first motion information of the current template includes determining the first motion information of the current template as motion information of a center position of the first collocated block; and
the determining the first reference template includes determining the first reference template associated with the first motion shift candidate based on the motion information of the center position of the first collocated block.

### Claim 20 (depends on 17)

The method of claim 17, wherein
the one of the first reference template and the first subblock reference template is the first subblock reference template;
the current template includes a first subblock template and a second subblock template;
the first motion information of the current template includes a first piece of motion information of the first subblock template and a second piece of motion information of the second subblock template that is different from the first piece of motion information;
the determining the first motion information of the current template includes determining the first piece of motion information of the first subblock template from a first subblock and determining the second piece of motion information of the second subblock template from a second subblock, the first subblock and the second subblock being different subblocks in the first collocated block;
the method further includes:
determining the first subblock reference template in the first reference picture based on the first piece of motion information of the first subblock template;
determining a second subblock reference template based on the second piece of motion information of the second subblock template, the second subblock reference template being in one of the first reference picture and in a second reference picture; and
determining a predictor template based on the first subblock reference template in the first reference picture and the second subblock reference template; and
the determining the first TM cost includes determining the first TM cost based on the current template and the predictor template.

### Claim 21

A method of processing visual media data, the method comprising:
processing a bitstream of the visual media data according to a format rule, wherein
the bitstream includes prediction information of a current block in a current picture from the bitstream, the prediction information indicating a subblock-based temporal motion vector prediction (SbTMVP) mode, and
the format rule specifies that template matching (TM) is performed by
determining a first collocated block in a first collocated picture based on a first motion shift candidate of the current block;
determining first motion information of a current template of the current block based on one or more pieces of motion information of the first collocated block, the first motion information of the current template indicating one of a first reference template and a first subblock reference template in a first reference picture relative to the current template that is in the current picture;
determining the one of the first reference template and the first subblock reference template associated with the first motion shift candidate based on the first motion information of the current template; and
determining a first TM cost based on the current template and the one of the first reference template and the first subblock reference template.
