Intra prediction for square and non-square blocks in video compression

A method for video encoding, comprising:

• for a current block which is a non-square block, determining whether an angular intra prediction mode for the current block is a wide angle mode that is in a direction outside a range of directions spanning a bottom left diagonal direction and a top right diagonal direction of the current block;
• when the angular intra prediction mode is determined to be the wide angle mode, determining whether a condition for applying an intra smoothing filter to blocks neighboring the current block is met, the condition being met when a block size of the non-square block is greater than or equal to a block size threshold;
• and
• when the condition is determined to be met,
• applying the intra smoothing filter to the blocks neighboring the current block;
• performing intra prediction with a coding tool based on the filtered blocks to encode the current block and to obtain a feature value for the current block;
• determining whether the coding tool used in the intra prediction is a Non-Separable Secondary Transform (NSST);
• and
• when the coding tool is determined to be the NSST, mapping the wide angle mode to a conventional intra prediction mode within the range of directions spanning the bottom left diagonal direction and the top right diagonal direction of the current block, the wide angle mode being mapped to an intra prediction mode in the range of directions that is nearest to the wide angle mode.

A non-transitory computer-readable storage medium storing instructions which when executed by a processor cause the processor to perform a method of encoding a bitstream comprising:

• for a current block which is a non-square block, determining whether an angular intra prediction mode for the current block is a wide angle mode that is in a direction outside a range of directions spanning a bottom left diagonal direction and a top right diagonal direction of the current block;
• when the angular intra prediction mode is determined to be the wide angle mode, determining whether a condition for applying an intra smoothing filter to blocks neighboring the current block is met, the condition being met when a block size of the non-square block is greater than or equal to a block size threshold;
• and
• when the condition is determined to be met,
• applying the intra smoothing filter to the blocks neighboring the current block;
• performing intra prediction with a coding tool based on the filtered blocks to encode the current block and to obtain a feature value for the current block;
• determining whether the coding tool used in the intra prediction is a Non-Separable Secondary Transform (NSST);
• and
• when the coding tool is determined to be the NSST, mapping the wide angle mode to a conventional intra prediction mode within the range of directions spanning the bottom left diagonal direction and the top right diagonal direction of the current block, the wide angle mode being mapped to an intra prediction mode in the range of directions that is nearest to the wide angle mode;
• and
• transmitting the bitstream including the encoded current block.