# US 12,010,302 B2 · Image processing apparatuses including CNN-based in-loop filter

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,010,302 B2
- Document ID: US12010302B2
- Status: Granted
- Kind: B2
- Application number: 18/088,615
- Filed: 2022-12-26
- Published: 2023-05-04
- Granted: 2024-06-11
- Earliest claimed: 2016-06-24
- Assignees: Korea Advanced Institute of Science and Technology
- Applicants: Korea Advanced Institute of Science and Technology
- Inventors: Mun Churl Kim
- Web page: https://patentgrep.com/patent/US12010302B2

## CPC

- H04N 19/117: Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635)
- G06F 17/15: Correlation function computation {including computation of convolution operations (arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00)}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- H04N 19/124: Quantisation
- H04N 19/159: Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
- H04N 19/172: the region being a picture, frame or field
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 19/30: using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence)
- H04N 19/82: involving filtering within a prediction loop
- H04N 19/86: involving reduction of coding artifacts, e.g. of blockiness
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed according to one exemplary embodiment includes not limited to: a filtering unit configured to generate filtering information by filtering a residual image corresponding to a difference between an original image and a prediction image; an inverse filtering unit configured to generate inverse filtering information by inversely filtering the filtering information; an estimator configured to generate the prediction image based on the original image and reconstruction information; a CNN-based in-loop filter configured to receive the inverse filtering information and the prediction image and to output the reconstruction information; and an encoder configured to perform encoding based on the filtering information and information of the prediction image, and wherein the CNN-based in-loop filter is trained for each of the plurality of artefact sections according to an artefact value or for each of the plurality of quantization parameter sections according to a quantization parameter.

## Claims

### Claim 1

An encoding apparatus comprising:
a filtering unit configured to generate filtering information by filtering a residual image corresponding to a difference between an original image and a prediction image;
an inverse filtering unit configured to generate inverse filtering information by inversely filtering the filtering information;
an estimator configured to generate the prediction image based on the original image and reconstruction information;
a convolutional neural network (CNN)-based in-loop filter configured to receive the inverse filtering information and the prediction image and to output the reconstruction information;
a decoded picture buffer configured to receive and store the reconstruction information; and
an encoder configured to perform encoding based on the filtering information and information of the prediction image,
wherein the CNN-based in-loop filter is trained for each of the plurality of artefact sections according to an artefact value or for each of the plurality of quantization parameter sections according to a quantization parameter.

### Claim 2 (depends on 1)

The encoding apparatus of claim 1, wherein the filtering unit is configured to generate filtering information by transforming and quantizing the residual image, and
the inverse filtering unit is configured to generate inverse filtering information by performing inverse quantization and inverse transformation on the filtering information.

### Claim 3 (depends on 1)

The encoding apparatus of claim 1, wherein the reconstruction information is in the same format as that of the original image, and
the CNN-based in-loop filter is configured to generate reconstruction information by inputting the inverse filtering information and prediction information based on the prediction image to the CNN-based in-loop filter.

### Claim 4 (depends on 3)

The encoding apparatus of claim 3, further comprising:
an in-loop filter configured to perform in-loop filtering on the prediction information.

### Claim 5 (depends on 4)

The encoding apparatus of claim 4, wherein the in-loop filter comprises at least one of a deblocking filter, a sample adaptive offset filter, and an adaptive loop filter.

### Claim 6 (depends on 1)

The encoding apparatus of claim 1, wherein the reconstruction information is in the same format as that of the residual image, and
the CNN-based in-loop filter is configured to generate reconstruction information by inputting the inverse filtering information and prediction information based on the prediction image to the CNN-based in-loop filter.

### Claim 7 (depends on 6)

The encoding apparatus of claim 6, further comprising:
an in-loop filter configured to perform in-loop filtering on the prediction information.

### Claim 8 (depends on 1)

The encoding apparatus of claim 1, further comprising:
an in-loop filter configured to perform in-loop filtering on the reconstruction information.

### Claim 9

A decoding apparatus comprising:
an entropy decoder configured to output filtering information and preliminary prediction information by decoding encoded bitstream information;
an inverse filtering unit configured to generate inverse filtering information by inversely filtering the filtering information;
an estimator configured to generate a prediction image based on the preliminary prediction information;
a convolutional neural network (CNN)-based in-loop filter configured to receive the inverse filtering information and the prediction image and to output reconstruction information; and
an encoded picture buffer configured to receive and store the reconstruction information,
wherein the CNN-based in-loop filter is trained for each of the plurality of artefact sections according to an artefact value or for each of the plurality of quantization parameter sections according to a quantization parameter.

### Claim 10 (depends on 9)

The decoding apparatus of claim 9, wherein the reconstruction information is in the same format as that of the original image, and
the CNN-based in-loop filter is configured to generate reconstruction information by inputting the inverse filtering information and prediction information based on the prediction image to the CNN-based in-loop filter.

### Claim 11 (depends on 10)

The decoding apparatus of claim 10, further comprising:
an in-loop filter configured to perform in-loop filtering on the inverse filtering information.

### Claim 12 (depends on 11)

The decoding apparatus of claim 11, wherein the in-loop filter comprises at least one of a deblocking filter, a sampled adaptive offset filter, and an adaptive loop filter.

### Claim 13 (depends on 9)

The decoding apparatus of claim 9, wherein the reconstruction information is in the same format as that of the residual image, and
the CNN-based in-loop filter is configured to generate reconstruction information by inputting the inverse filtering information and prediction information based on the prediction image to the CNN-based in-loop filter.

### Claim 14 (depends on 13)

The decoding apparatus of claim 13, further comprising:
an adder configured to generate final reconstruction information by adding the reconstruction information and the prediction image.

### Claim 15 (depends on 13)

The decoding apparatus of claim 13, further comprising:
an in-loop filter configured to perform in-loop filtering on the inverse filtering information.

### Claim 16 (depends on 15)

The decoding apparatus of claim 15, wherein the in-loop filter comprises at least one of a deblocking filter, a sample adaptive offset filter, and an adaptive loop filter.

### Claim 17 (depends on 9)

The decoding apparatus of claim 9, wherein the reconstruction information is in the same format as that of the residual image, and
the CNN-based in-loop filter is configured to generate residual reconstruction information by inputting the inverse filtering information to the CNN-based in-loop filter.

### Claim 18 (depends on 17)

The decoding apparatus of claim 17, further comprising:
an adder configured to generate final reconstruction information by adding the residual reconstruction information and the prediction image.

### Claim 19 (depends on 18)

The decoding apparatus of claim 18, further comprising:
an in-loop filter configured to perform in-loop filtering on the final reconstruction information.

### Claim 20 (depends on 19)

The decoding apparatus of claim 19, wherein the in-loop filter comprises at least one of a deblocking filter, a sample adaptive offset filter, and an adaptive loop filter.