Matrix derivation in intra coding mode

An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:

• determine, for a first conversion between a first video block of the video and a bitstream of the video, that a first intra mode is applied on the first video block of the video;
• derive reference samples of the first video block;
• perform a boundary downsampling operation on the reference samples of the first video block based on a size of the first video block, followed by a matrix vector multiplication operation, and selectively followed by an upsampling operation to generate prediction samples for the first video block;
• and
• perform the first conversion based on the prediction samples of the first video block,
• wherein elements of matrices or offset values applied in the matrix vector multiplication operation have a fixed precision,
• wherein the boundary downsampling operation includes deriving, according to a rule, boundary samples by applying a right bit shift operation on a sum of at least one reference sample in response to a number of shifted bits being greater than zero,
• wherein the boundary samples redBdryS[x] are calculated using one of following equations:
• redBdryS[x]=(Σi=0bDwn−1 refS[x*bDwn+i]+(1<<(Log 2(bDwn)−1)))>>Log 2(bDwn), if bDwn>1, or
• redBdryS[x]=refS[x] if bDwn=1,
• wherein bDwn is equal to a function of the size of the first video block and a boundary size,
• wherein refS [x] indicates a number of reference samples x,
• wherein >>indicates the right bit shift operation, and
• wherein <<indicates a left bit shift operation.

A non-transitory computer-readable storage medium storing instructions that cause a processor to:

• determine, for a first conversion between a first video block of the video and a bitstream of the video, that a first intra mode is applied on the first video block of the video;
• derive reference samples of the first video block;
• perform a boundary downsampling operation on the reference samples of the first video block based on a size of the first video block, followed by a matrix vector multiplication operation, and selectively followed by an upsampling operation to generate prediction samples for the first video block;
• and
• perform the first conversion based on the prediction samples of the first video block,
• wherein elements of matrices or offset values applied in the matrix vector multiplication operation have a fixed precision,
• wherein the boundary downsampling operation includes deriving, according to a rule, boundary samples by applying a right bit shift operation on a sum of at least one reference sample in response to a number of shifted bits being greater than zero,
• wherein the boundary samples redBdryS[x] are calculated using one of following equations:
• redBdryS[x]=(Σi=0bDwn−1 refS[x*bDwn+i]+(1<<(Log 2(bDwn)−1)))>>Log 2(bDwn), if bDwn>1, or
• redBdryS[x]=refS[x] if bDwn=1,
• wherein bDwn is equal to a function of the size of the first video block and a boundary size,
• wherein refS [x] indicates a number of reference samples x,
• wherein >>indicates the right bit shift operation, and
• wherein <<indicates a left bit shift operation.

A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises:

• determining that a first intra mode is applied on a first video block of the video;
• deriving reference samples of the first video block;
• performing a boundary downsampling operation on the reference samples of the first video block based on a size of the first video block, followed by a matrix vector multiplication operation, and selectively followed by an upsampling operation to generate prediction samples for the first video block;
• and
• generating the bitstream based on the prediction samples of the first video block;
• wherein elements of matrices or offset values applied in the matrix vector multiplication operation have a fixed precision,
• wherein the boundary downsampling operation includes deriving, according to a rule, boundary samples by applying a right bit shift operation on a sum of at least one reference sample in response to a number of shifted bits being greater than zero,
• wherein the boundary samples redBdryS[x] are calculated using one of following equations:
• redBdryS[x]=(Σi=0bDwn−1 refS[x*bDwn+i]+(1<<(Log 2(bDwn)−1)))>>Log 2(bDwn), if bDwn>1, or
• redBdryS[x]=refS[x] if bDwn=1,
• wherein bDwn is equal to a function of the size of the first video block and a boundary size,
• wherein refS [x] indicates a number of reference samples x,
• wherein >>indicates the right bit shift operation, and
• wherein <<indicates a left bit shift operation.