# US 2020/0404325 A1 · Signaling of matrix intra prediction parameters in video coding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0404325 A1
- Document ID: US20200404325A1
- Status: Published application
- Kind: A1
- Application number: 16/905,182
- Filed: 2020-06-18
- Published: 2020-12-24
- Earliest claimed: 2019-06-21
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Adarsh Krishnan Ramasubramonian; Geert Van der Auwera; Cheng-Teh Hsieh; Thibaud Laurent Biatek; Luong Pham Van; Marta Karczewicz
- Web page: https://patentgrep.com/patent/US20200404325A1

## CPC

- H04N 19/593: involving spatial prediction techniques
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N 19/11: among a plurality of spatial predictive coding modes
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/44: Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A video coder determines a plurality of available Matrix Intra Prediction (MIP) parameter sets (MPS's) for a picture of video data. The plurality of available MPS's is a union of (i) a subset of all default MPS's and (ii) a set of additional MPS's that are signaled in the bitstream. Each of the default MPS's is associated with a predefined MIP mode in a codec. Each of the set of additional MPS's is associated with a new MIP mode in a set of new MIP modes. The video decoder uses a MIP mode associated with an MPS in the plurality of available MPS's to generate a prediction block for a current block of the picture.

## Claims

### Claim 1

A method of decoding video data, the method comprising:
obtaining a bitstream that includes an encoded representation of the video data that is encoded using a codec;
determining a plurality of available Matrix Intra Prediction (MIP) parameter sets (MPS's) for a picture of the video data, wherein:
the plurality of available MPS's is a union of (i) a subset of all default MPS's and (ii) a set of additional MPS's that are signaled in the bitstream,
each of the default MPS's is associated with a predefined MIP mode in the codec, and
each of the set of additional MPS's is associated with a new MIP mode in a set of new MIP modes;
using a MIP mode associated with an MPS in the plurality of available MPS's to generate a prediction block for a current block of the picture; and
reconstructing the current block based on the prediction block for the current block and residual data for the current block.

### Claim 2 (depends on 1)

The method of claim 1, wherein:
the plurality of available MPS's includes at least one MPS for each category in a plurality of categories, and
each category in the plurality of categories corresponds to a different set of block sizes.

### Claim 3 (depends on 1)

The method of claim 1, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode,
a syntax element in the bitstream indicates whether the MIP mode is specified with respect to an already-specified MIP mode or whether the MIP mode is specified independently, and
the method further comprises, based on the syntax element indicating that the MIP mode is specified with respect to the already-specified mode, deriving parameters of the MIP mode using parameters of one or more MIP modes.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode,
a predicted sample location array (PSLA) of the MIP mode is a set of sample locations in the current block that correspond to locations where the MIP mode directly generates predicted values,
the MPS associated with the MIP mode includes a list of explicit sample predictor parameter subsets (ESPPS's) for the MIP mode,
the list of ESPPS's for the MIP mode is signaled in the bitstream,
each of the ESPPS's corresponds to a sample location in the PSLA of the MIP mode, and
using the MIP mode to generate the prediction block for the current block comprises:
for each sample location in the PSLA of the MIP mode, predicting a sample for the sample location using parameters contained in the ESPPS corresponding to the sample location; and
interpolating remaining samples of the prediction block based on the predicted samples for the sample locations of the PSLA of the MIP mode.

### Claim 5 (depends on 4)

The method of claim 4, wherein the ESPPS's for the MIP mode are signaled in the bitstream dependent on another ESPPS.

### Claim 6 (depends on 5)

The method of claim 5, wherein the other ESPPS is one of another E-MIP mode or a particular predefined MIP mode in the codec.

### Claim 7 (depends on 1)

The method of claim 1, further comprising determining, based on a syntax element signaled in a sequence parameter set (SPS) of the bitstream, that a particular default MPS of the default MPS's is available to be used for prediction by pictures of the video data that refer to the SPS.

### Claim 8 (depends on 1)

The method of claim 1, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode,
a syntax element in the bitstream indicates whether the MIP mode is specified with respect to an already-specified MIP mode or whether the MIP mode is specified independently, and
the method further comprises, based on the syntax element indicating that the E-MIP mode is specified independently, deriving parameters of the MIP mode without using any parameter of any other MIP mode.

### Claim 9 (depends on 1)

The method of claim 1, wherein:
a MIP mode index is signaled in the bitstream and signaling of the MIP mode index is dependent on a number of MIP modes in a MIP mode list, and
the MIP mode index indicates the MIP mode.

### Claim 10 (depends on 1)

The method of claim 1, wherein:
a reference sample location array (RSLA) for the MIP mode is signaled in the bitstream, the RSLA for the MIP mode indicating a set of reference sample locations in a neighborhood of the current block that are used in the MIP mode, and
using the MIP mode to generate the prediction block comprises:
for each sample location in a predicted sample location array (PSLA) of the MIP mode, determining a prediction value by multiplying a reference sample at a reference sample location indicated by the RSLA for the MIP mode by a matrix coefficient of a matrix specified in the MPS associated with the MIP mode and adding an offset value specified in the MPS associated with the MIP mode.

### Claim 11

A method of encoding video data using a codec, the method comprising:
determining a plurality of available Matrix Intra Prediction (MIP) parameter sets (MPS's) for a picture of the video data, wherein:
the plurality of available MPS's is a union of (i) a subset of all default MPS's and (ii) a set of additional MPS's that are signaled in the bitstream,
each of the default MPS's is associated with a predefined MIP mode in the codec, and
each of the set of additional MPS's is associated with a new MIP mode;
signaling the set of additional MPS's in a bitstream that comprises an encoded representation of the video data;
using a MIP mode associated with an MPS in the plurality of available MPS's to generate a prediction block for a current block of the picture; and
generating residual data for the current block based on the prediction block for the current block and samples of the current block.

### Claim 12 (depends on 11)

The method of claim 11, wherein:
the plurality of available MPS's includes at least one MPS for each category in a plurality of categories, and
each category in the plurality of categories correspond to a different set of block sizes.

### Claim 13 (depends on 11)

The method of claim 11, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode, and
the method further comprises:
deriving parameters of the MIP mode using parameters of one or more MIP modes; and
signaling a syntax element in the bitstream indicates that the MIP mode is specified with respect to an already-specified MIP mode.

### Claim 14 (depends on 11)

The method of claim 11, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode,
a predicted sample location array (PSLA) of the MIP mode is a set of sample locations in the current block that correspond to locations where the MIP mode directly generates predicted values,
the MPS associated with the MIP mode includes a list of explicit sample predictor parameter subsets (ESPPS's) for the MIP mode,
the method further comprises signaling the list of ESPPS's for the MIP mode in the bitstream,
each of the ESPPS's corresponds to a sample location in the PSLA of the MIP mode, and
using the MIP mode to generate the prediction block for the current block comprises:
for each sample location in the PSLA of the MIP mode, predicting a sample for the sample location using parameters contained in the ESPPS corresponding to the sample location; and
interpolating remaining samples of the prediction block based on the predicted samples for the sample locations of the PSLA of the MIP mode.

### Claim 15 (depends on 14)

The method of claim 14, wherein the method further comprises signaling the ESPPS's for the MIP mode in the bitstream dependent on another ESPPS.

### Claim 16 (depends on 15)

The method of claim 15, wherein the other ESPPS is one of another E-MIP mode or a particular predefined MIP mode in the codec.

### Claim 17 (depends on 11)

The method of claim 11, further comprising including, in a sequence parameter set (SPS) of the bitstream, a syntax element that indicates whether a particular default MPS of the default MPS's is available to be used for prediction by pictures of the video data that refer to the SPS.

### Claim 18 (depends on 11)

The method of claim 11, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode,
the method further comprises:
deriving parameters of the MIP mode without using any parameter of any other MIP mode; and
signaling a syntax element in the bitstream that indicates that the MIP mode is specified with independently.

### Claim 19 (depends on 11)

The method of claim 11, wherein the method further comprises signaling a MIP mode index that indicates the MIP mode, wherein signaling of the MIP mode index is dependent on a number of MIP modes in a MIP mode list.

### Claim 20 (depends on 11)

The method of claim 11, wherein:
the method further comprises signaling a reference sample location array (RSLA) for the MIP mode in the bitstream, the RSLA for the MIP mode indicating a set of reference sample locations in a neighborhood of the current block that are used in the MIP mode, and
using the MIP mode to generate the prediction block comprises:
for each sample location in a predicted sample location array (PSLA) of the MIP mode, determining a prediction value by multiplying a reference sample at a reference sample location indicated by the RSLA for the MIP mode by a matrix coefficient of a matrix specified in the MPS associated with the MIP mode and adding an offset value specified in the MPS associated with the MIP mode.

### Claim 21

A device for decoding video data, the device comprising:
a memory to store the video data; and
one or more processors implemented in circuitry, the one or more processors configured to:
obtain a bitstream that includes an encoded representation of the video data that is encoded using a codec;
determine a plurality of available Matrix Intra Prediction (MIP) parameter sets (MPS's) for a picture of the video data, wherein:
the plurality of available MPS's is a union of (i) a subset of all default MPS's and (ii) a set of additional MPS's that are signaled in the bitstream,
each of the default MPS's is associated with a predefined MIP mode in the codec, and
each of the set of additional MPS's is associated with a new MIP mode in a set of new MIP modes;
use a MIP mode associated with an MPS in the plurality of available MPS's to generate a prediction block for a current block of the picture; and
reconstruct the current block based on the prediction block for the current block and residual data for the current block.

### Claim 22 (depends on 21)

The device of claim 21, wherein:
the plurality of available MPS's includes at least one MPS for each category in a plurality of categories, and
each category in the plurality of categories correspond to a different set of block sizes.

### Claim 23 (depends on 21)

The device of claim 21, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode,
a syntax element in the bitstream indicates whether the MIP mode is specified with respect to an already-specified MIP mode or whether the MIP mode is specified independently, and
the one or more processors are further configured to derive, based on the syntax element indicating that the MIP mode is specified with respect to the already-specified mode, parameters of the MIP mode using parameters of one or more MIP modes.

### Claim 24 (depends on 21)

The device of claim 21, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode,
a predicted sample location array (PSLA) of the MIP mode is a set of sample locations in the current block that correspond to locations where the MIP mode directly generates predicted values,
the MPS associated with the MIP mode includes a list of explicit sample predictor parameter subsets (ESPPS's) for the MIP mode,
the list of ESPPS's for the MIP mode is signaled in the bitstream,
each of the ESPPS's corresponds to a sample location in the PSLA of the MIP mode, and
the one or more processors are configured such that, as part of using the MIP mode to generate the prediction block for the current block, the one or more processors:
for each sample location in the PSLA of the MIP mode, predict a sample for the sample location using parameters contained in the ESPPS corresponding to the sample location; and
interpolate remaining samples of the prediction block based on the predicted samples for the sample locations of the PSLA of the MIP mode.

### Claim 25 (depends on 24)

The device of claim 24, wherein the ESPPS's for the MIP mode are signaled in the bitstream dependent on another ESPPS.

### Claim 26 (depends on 25)

The device of claim 25, wherein the other ESPPS is one of another E-MIP mode or a particular predefined MIP mode in the codec.

### Claim 27 (depends on 21)

The device of claim 21, wherein the one or more processors are further configured to determine, based on a syntax element signaled in a sequence parameter set (SPS) of the bitstream, that a particular default MPS of the default MPS's is available to be used for prediction by pictures of the video data that refer to the SPS.

### Claim 28 (depends on 21)

The device of claim 21, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode,
a syntax element in the bitstream indicates whether the MIP mode is specified with respect to an already-specified MIP mode or whether the MIP mode is specified independently, and
the one or more processors are further configured to derive, based on the syntax element indicating that the MIP mode is specified independently, parameters of the MIP mode without using any parameter of any other MIP mode.

### Claim 29 (depends on 21)

The device of claim 21, wherein:
a MIP mode index is signaled in the bitstream and signaling of the MIP mode index is dependent on a number of MIP modes in a MIP mode list, and
the MIP mode index indicates the MIP mode.

### Claim 30 (depends on 21)

The device of claim 21, wherein:
a reference sample location array (RSLA) for the MIP mode is signaled in the bitstream, the RSLA for the MIP mode indicating a set of reference sample locations in a neighborhood of the current block that are used in the MIP mode, and
the one or more processors are configured such that, as part of using the MIP mode to generate the prediction block, the one or more processors:
for each sample location in a predicted sample location array (PSLA) of the MIP mode, determine a prediction value by multiplying a reference sample at a reference sample location indicated by the RSLA for the MIP mode by a matrix coefficient of a matrix specified in the MPS associated with the MIP mode and adding an offset value specified in the MPS associated with the MIP mode.

### Claim 31 (depends on 21)

The device of claim 21, further comprising a display configured to display decoded video data.

### Claim 32 (depends on 21)

The device of claim 21, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

### Claim 33

A device for encoding video data using a codec, the device comprising:
a memory to store the video data; and
one or more processors implemented in circuitry, the one or more processors configured to:
determine a plurality of available Matrix Intra Prediction (MIP) parameter sets (MPS's) for a picture of the video data, wherein:
the plurality of available MPS's is a union of (i) a subset of all default MPS's and (ii) a set of additional MPS's that are signaled in the bitstream,
each of the default MPS's is associated with a predefined MIP mode in the codec, and
each of the set of additional MPS's is associated with a new MIP mode;
signaling the set of additional MPS's in a bitstream that comprises an encoded representation of the video data;
use a MIP mode associated with an MPS in the plurality of available MPS's to generate a prediction block for a current block of the picture; and
generate residual data for the current block based on the prediction block for the current block and samples of the current block.

### Claim 34 (depends on 33)

The device of claim 33, wherein:
the plurality of available MPS's includes at least one MPS for each category in a plurality of categories, and
each category in the plurality of categories correspond to a different set of block sizes.

### Claim 35 (depends on 33)

The device of claim 33, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode, and
the one or more processors are further configured to:
derive parameters of the MIP mode using parameters of one or more MIP modes; and
signal a syntax element in the bitstream indicates that the MIP mode is specified with respect to an already-specified MIP mode.

### Claim 36 (depends on 33)

The device of claim 33, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode,
a predicted sample location array (PSLA) of the MIP mode is a set of sample locations in the current block that correspond to locations where the MIP mode directly generates predicted values,
the MPS associated with the MIP mode includes a list of explicit sample predictor parameter subsets (ESPPS's) for the MIP mode,
the one or more processors are further configured to signal the list of ESPPS's for the MIP mode in the bitstream,
each of the ESPPS's corresponds to a sample location in the PSLA of the MIP mode, and
the one or more processors are configured such that, as part of using the MIP mode to generate the prediction block for the current block, the one or more processors:
for each sample location in the PSLA of the MIP mode, predict a sample for the sample location using parameters contained in the ESPPS corresponding to the sample location; and
interpolate remaining samples of the prediction block based on the predicted samples for the sample locations of the PSLA of the MIP mode.

### Claim 37 (depends on 36)

The device of claim 36, wherein the one or more processors are further configured to signal the ESPPS's for the MIP mode in the bitstream dependent on another ESPPS.

### Claim 38 (depends on 37)

The device of claim 37, wherein the other ESPPS is one of another E-MIP mode or a particular predefined MIP mode in the codec.

### Claim 39 (depends on 33)

The device of claim 33, wherein the one or more processors are further configured to include, in a sequence parameter set (SPS) of the bitstream, a syntax element that indicates whether a particular default MPS of the default MPS's is available to be used for prediction by pictures of the video data that refer to the SPS.

### Claim 40 (depends on 33)

The device of claim 33, wherein:
the MIP mode is in the set of new MIP modes and is an explicit MIP (E-MIP) mode,
the one or more processors are further configured to:
derive parameters of the MIP mode without using any parameter of any other MIP mode; and
signal a syntax element in the bitstream that indicates that the MIP mode is specified with independently.

### Claim 41 (depends on 33)

The device of claim 33, wherein the one or more processors are further configured to signal a MIP mode index that indicates the MIP mode, wherein signaling of the MIP mode index is dependent on a number of MIP modes in a MIP mode list.

### Claim 42 (depends on 33)

The device of claim 33, wherein:
the one or more processors are further configured to signal a reference sample location array (RSLA) for the MIP mode in the bitstream, the RSLA for the MIP mode indicating a set of reference sample locations in a neighborhood of the current block that are used in the MIP mode, and
the one or more processors are configured such that, as part of using the MIP mode to generate the prediction block, the one or more processors:
for each sample location in a predicted sample location array (PSLA) of the MIP mode, determine a prediction value by multiplying a reference sample at a reference sample location indicated by the RSLA for the MIP mode by a matrix coefficient of a matrix specified in the MPS associated with the MIP mode and adding an offset value specified in the MPS associated with the MIP mode.

### Claim 43 (depends on 33)

The device of claim 33, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.
