# US 2023/0057982 A1 · Matrix-based intra prediction using filtering

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0057982 A1
- Document ID: US20230057982A1
- Status: Published application
- Kind: A1
- Application number: 17/894,878
- Filed: 2022-08-24
- Published: 2023-02-23
- Earliest claimed: 2019-05-15
- Assignees: Beijing Bytedance Network Technology Co., Ltd.; Bytedance Inc.
- Applicants: Beijing Bytedance Network Technology Co., Ltd.; Bytedance Inc.
- Inventors: Zhipin Deng; Kai Zhang; Li Zhang; Hongbin Liu; Jizheng Xu
- Web page: https://patentgrep.com/patent/US20230057982A1

## CPC

- H04N 19/593: involving spatial prediction techniques
- H04N 19/105: Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
- H04N 19/11: among a plurality of spatial predictive coding modes
- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/157: Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter
- H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/184: the unit being bits, e.g. of the compressed video stream
- H04N 19/70: characterised by syntax aspects related to video coding, e.g. related to compression standards
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

Devices, systems and methods for digital video coding, which includes matrix-based intra prediction methods for video coding, are described. In a representative aspect, a method for video processing includes performing a conversion between a current video block of a video and a bitstream representation of the current video block using a matrix based intra prediction (MIP) mode in which a prediction block of the current video block is determined by performing, on reference boundary samples located to a left of the current video block and located to a top of the current video block, a boundary downsampling operation, followed by a matrix vector multiplication operation, and selectively followed by an upsampling operation, where instead of reduced boundary samples calculated from the reference boundary samples of the current video block in the boundary downsampling operation, the reference boundary samples are directly used for a prediction process in the upsampling operation.

## Claims

### Claim 1

A method of processing video data, comprising:
determining, for a conversion between a video block of a video and a bitstream of the video, that a first intra mode is applied on the video block of the video, wherein process in the first intra mode includes a one-stage boundary downsampling operation, followed by a matrix vector multiplication operation and an upsampling operation to generate prediction samples for the video block of the video; and
performing the conversion based on the prediction samples;
wherein reduced boundary samples are generated from reference boundary samples of the video block of the video by the one-stage boundary downsampling operation and are used to generate inputs to the matrix vector multiplication operation, and
wherein the reference boundary samples include left and above reference boundary samples of the video block.

### Claim 2 (depends on 1)

The method of claim 1, wherein the left and above reference boundary samples are derived without an intra reference sample filtering process.

### Claim 3 (depends on 1)

The method of claim 1, wherein in the one-stage boundary downsampling operation, the reduced boundary samples are generated directly from the reference boundary samples of the video block and a downscaling factor without deriving intermediate samples, and
wherein the downscaling factor is calculated only once for a horizontal direction and a vertical direction, respectively.

### Claim 4 (depends on 3)

The method of claim 3, wherein the downscaling factor is calculated based on a size of the video block.

### Claim 5 (depends on 1)

The method of claim 1, wherein inputting samples to the upsampling operation include upsampling boundary samples of the video block, and the upsampling boundary samples are not computed by averaging the reference boundary samples of the video block.

### Claim 6 (depends on 5)

The method of claim 5, wherein at least one of the upsampling boundary samples are copied from the reference boundary samples.

### Claim 7 (depends on 1)

The method of claim 1, wherein a first syntax element indicating whether to apply the first intra mode is included in the bitstream.

### Claim 8 (depends on 1)

The method of claim 1, wherein the matrix vector multiplication operation is followed by a transposing operation prior to the upsampling operation.

### Claim 9 (depends on 8)

The method of claim 8, wherein the transposing operation converts a block having a width of a first value and a height of a second value to a block having the width of the second value and a height of the first value.

### Claim 10 (depends on 1)

The method of claim 1, wherein the conversion includes encoding the video block into the bitstream.

### Claim 11 (depends on 1)

The method of claim 1, wherein the conversion includes decoding the video block from the bitstream.

### Claim 12

An apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor, cause the processor to:
determine, for a conversion between a video block of a video and a bitstream of the video, that a first intra mode is applied on the video block of the video, wherein process in the first intra mode includes a one-stage boundary downsampling operation, followed by a matrix vector multiplication operation and an upsampling operation to generate prediction samples for the video block of the video; and
perform the conversion based on the prediction samples;
wherein reduced boundary samples are generated from reference boundary samples of the video block of the video by the one-stage boundary downsampling operation and are used to generate inputs to the matrix vector multiplication operation, and
wherein the reference boundary samples include left and above reference boundary samples of the video block.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the left and above reference boundary samples of the video block are derived without an intra reference sample filtering process.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein in the one-stage boundary downsampling operation, the reduced boundary samples are generated directly from the reference boundary samples of the video block and a downscaling factor without deriving intermediate samples, and
wherein the downscaling factor is calculated only once for a horizontal direction and a vertical direction, respectively.

### Claim 15 (depends on 14)

The apparatus of claim 14, wherein the downscaling factor is calculated based on a size of the video block.

### Claim 16 (depends on 12)

The apparatus of claim 12, wherein inputting samples to the upsampling operation include upsampling boundary samples of the video block, wherein at least one of the upsampling boundary samples are copied from the reference boundary samples.

### Claim 17 (depends on 12)

The apparatus of claim 12, wherein inputting samples to the upsampling operation include upsampling boundary samples of the video block, and the upsampling boundary samples are not computed by averaging the reference boundary samples of the video block.

### Claim 18

A non-transitory computer-readable storage medium storing instructions that cause a processor to:
determine, for a conversion between a video block of a video and a bitstream of the video, that a first intra mode is applied on the video block of the video, wherein process in the first intra mode includes a one-stage boundary downsampling operation, followed by a matrix vector multiplication operation and an upsampling operation to generate prediction samples for the video block of the video; and
perform the conversion based on the prediction samples;
wherein reduced samples are generated from reference boundary samples of the video block of the video by the one-stage boundary downsampling operation and are used to generate inputs to the matrix vector multiplication operation, and
wherein the reference boundary samples include left and above reference boundary samples of the video block.

### Claim 19 (depends on 18)

The non-transitory computer-readable storage medium of claim 18, wherein the left and above reference boundary samples are derived without an intra reference sample filtering process.

### Claim 20

A non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises:
determining, that a first intra mode is applied on a video block of the video, wherein process in the first intra mode includes a one-stage boundary downsampling operation, followed by a matrix vector multiplication operation and an upsampling operation to generate prediction samples for the video block of the video; and
generating the bitstream based on the determining;
wherein reduced samples are generated from reference boundary samples of the video block of the video by the one-stage boundary downsampling operation and are used to generate inputs to the matrix vector multiplication operation, and
wherein the reference boundary samples include left and above reference boundary samples of the video block.
