Method and apparatus for selecting transform selection in an encoder and decoder

A decoder comprising:

• at least one processor;
• at least one memory coupled to the at least one processor, said at least one memory comprising instructions executable by the at least one processor, which cause the at least one processor to perform operations comprising:
• receiving an encoded video block having at least one flag encoded using context-based adaptive arithmetic coding;
• parsing the at least one flag to determine if the at least one flag is set to signal that a first transform of a plurality of transforms is to be used to decode the encoded video block in both a horizontal direction and a vertical direction;
• responsive to the at least one flag being set to signal that the first transform is to be used in both the horizontal direction and in the vertical direction:
• decoding the encoded video block in the horizontal direction and the vertical direction using the first transform to generate a decoded video block;
• responsive to the at least one flag being set to signal that the first transform is not to be used in both the horizontal direction and in the vertical direction:
• parsing a second flag of the at least one flag to determine if the second flag is set to signal a second transform of the plurality of transforms is to be used to decode the encoded video block in both a horizontal direction and a vertical direction to generate the decoded video block;
• responsive to the at least one flag being set to signal that the first transform is to be used in both the horizontal direction and in the vertical direction:
• decoding the encoded video block in the horizontal direction and the vertical direction using the second transform to generate the decoded video block;
• responsive to the at least one flag being set to signal that the first transform is not to be used in both the horizontal direction and in the vertical direction:
• parsing a third flag of the at least one flag to determine in which of the horizontal direction or the vertical direction the second transform is to be used to decode the encoded video block and in which of the horizontal direction or the vertical direction a third transform is to be used to decode the encoded video block;
• and
• decoding the encoded video block using the second and third transforms to generate the decoded video block.

A method performed by an encoder, the method comprising:

• receiving a video block for encoding;
• determining a characteristic of the video block;
• responsive to the characteristic being of a type that indicates a multiple transform selection is used:
• selecting a first transform in a plurality of transforms that is part of the multiple transform selection and that is one of most computationally expensive to use or least likely to be used in encoding the video block;
• testing combinations of the plurality of transforms in a horizontal direction and a vertical direction without testing a combination where the first transform is used in both the horizontal direction and the vertical direction;
• selecting a combination from the combinations that provides the lowest rate distortion;
• encoding the video block using the selected combination to generate an encoded video block;
• responsive to the characteristic being of a type that indicates a multiple transform selection is not to be used:
• encoding the video block using a default transform in the horizontal direction and the vertical direction.

An encoder for encoding a block of video based on a block size of the block, wherein each of a horizontal direction and a vertical direction of the block is encoded using a transform, wherein the transform can be one of a first transform, a second transform or a third transform, the encoder comprising:

• at least one processor;
• at least one memory coupled to the at least one processor, said at least one memory comprising instructions executable by the at least one processor, which cause the at least one processor to perform operations comprising:
• determining a characteristic of the video block;
• responsive to the characteristic being of a type that indicates a multiple transform selection is used:
• selecting a first transform in a plurality of transforms that is part of the multiple transform selection and that is one of most computationally expensive to use or least likely to be used in encoding the video block;
• testing combinations of the plurality of transforms in a horizontal direction and a vertical direction without testing a combination where the first transform is used in both the horizontal direction and the vertical direction;
• selecting a combination from the combinations that provides the lowest rate distortion;
• encoding the video block using the selected combination to generate an encoded video block;
• responsive to the characteristic being of a type that indicates a multiple transform selection is not to be used:
• encoding the video block using a default transform in the horizontal direction and the vertical direction.