Determination of intra prediction mode for indexation into non-separable transform kernels

A method of encoding by an encoder, comprising:

• in response to at least one non-separable transform being enabled for an intra prediction method, determining, by the processor, an intra prediction mode and the at least one non-separable transform for use in encoding a coding unit (CU) to a bitstream, the at least one non-separable transform being associated with a plurality of transform matrix sets;
• selecting, by the processor, a transform matrix from the plurality of transform matrix sets based on a bitstream index and the intra prediction mode;
• and
• encoding, by the processor, the CU based on the intra prediction mode and the transform matrix,
• wherein the intra prediction mode is determined in response to one or more of intra block copy (IBC), matrix weighted intra prediction (MIP), or intra template matching prediction (IntraTMP) being used for the CU, and
• wherein the at least one non-separable transform includes one or more of a low frequency non-separable secondary transform (LFNST) or a non-separable primary transform (NSPT).

A non-transitory computer-readable medium storing instructions, when executed by a processor, cause the processor to perform the following operations to generate a bitstream and store the bitstream:

• in response to at least one non-separable transform being enabled for an intra prediction method, determine an intra prediction mode and the at least one non-separable transform for use in encoding a coding unit (CU) to the bitstream, the at least one non-separable transform being associated with a plurality of transform matrix sets;
• select a transform matrix from the plurality of transform matrix sets based on a bitstream index and the intra prediction mode;
• and
• encode the CU based on the intra prediction method and the transform matrix,
• wherein the intra prediction mode is determined in response to one or more of intra block copy (IBC), matrix weighted intra prediction (MIP), or intra template matching prediction (IntraTMP) being used for the CU, and
• wherein the at least one non-separable transform includes one or more of a low frequency non-separable secondary transform (LFNST) or a non-separable primary transform (NSPT).