Systems and methods for applying non-separable transforms on inter prediction residuals

A method of video encoding performed at a computing system having memory and one or more processors, the method comprising:

• receiving video data that includes a current block to be encoded in an inter mode;
• deriving a set of one or more inter mode residual blocks for the current block;
• generating a set of transformed residual blocks by applying a primary transform to the set of one or more inter mode residual blocks;
• when the primary transform applied to the set of one or more inter mode residual blocks is a DCT-DCT transform, identifying an intra secondary transform (IST) kernel from a default set of inter IST kernels;
• generating a refined set of transform coefficients by applying the IST kernel to the set of transformed residual blocks;
• determining a value for an indicator according to the IST kernel;
• signaling the indicator in a video bitstream;
• and
• signaling the refined set of transform coefficients in the video bitstream.

A method of generating a video bitstream, the method comprising:

• generating the video bitstream, including:
• deriving a set of one or more inter mode residual blocks for a current block of video data;
• generating a set of transformed residual blocks by applying a primary transform to the set of one or more inter mode residual blocks;
• when the primary transform applied to the set of one or more inter mode residual blocks is a DCT-DCT transform, identifying an intra secondary transform (IST) kernel from a default set of inter IST kernels;
• generating a refined set of transform coefficients by applying the IST kernel to the set of transformed residual blocks;
• and
• determining a value for an indicator according to the IST kernel;
• and
• transmitting the video bitstream including the indicator and the refined set of transform coefficients in the video bitstream.