# US 2021/0176490 A1 · Image encoding method and apparatus and image decoding method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0176490 A1
- Document ID: US20210176490A1
- Status: Published application
- Kind: A1
- Application number: 17/181,744
- Filed: 2021-02-22
- Published: 2021-06-10
- Earliest claimed: 2019-10-29
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Jongseok LEE; Youngo PARK; Pilkyu PARK; Yumi SOHN
- Web page: https://patentgrep.com/patent/US20210176490A1

## CPC

- H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 3/4046: using neural networks
- H04N 19/172: the region being a picture, frame or field
- H04N 19/61: in combination with predictive coding
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Methods and apparatuses for image encoding and image decoding are provided. The image decoding method includes: obtaining deep neural network (DNN) update permission information indicating whether one or more pieces of DNN setting information are updated; based on the DNN update permission information indicating that the one or more pieces of the DNN setting information are updated, obtaining DNN update information necessary for determining one or more pieces of the DNN setting information that are updated; determining the one or more pieces of the updated DNN setting information according to the DNN update information; and obtaining a third image by performing artificial intelligence (AI) up-scaling on a second image according to the one or more pieces of the updated DNN setting information.

## Claims

### Claim 1

An image decoding method comprising:
obtaining deep neural network (DNN) update permission information indicating whether one or more pieces of DNN setting information are updated;
based on the DNN update permission information indicating that the one or more pieces of the DNN setting information are updated, obtaining DNN update information necessary for determining one or more pieces of the DNN setting information that are updated;
determining the one or more pieces of the updated DNN setting information according to the DNN update information; and
obtaining a third image by performing artificial intelligence (AI) up-scaling on a second image according to the one or more pieces of the updated DNN setting information.

### Claim 2 (depends on 1)

The image decoding method of claim 1, further comprising, based on the DNN update permission information indicating that the one or more pieces of DNN setting information are not updated, obtaining the third image by performing the AI up-scaling on the second image according to one or more pieces of default DNN setting information.

### Claim 3 (depends on 1)

The image decoding method of claim 1, wherein the DNN update information comprises DNN number information indicating a number of the one or more pieces of the updated DNN setting information, and
wherein the determining the one or more pieces of the updated DNN setting information comprises determining whether the number of the one or more pieces of the updated DNN setting information matches the number of the one or more pieces of the updated DNN setting information indicated by the DNN number information.

### Claim 4 (depends on 1)

The image decoding method of claim 1, wherein the DNN update information further comprises at least one of layer number information indicating a number of layers included in a DNN, input channel information indicating a number of input channels in each of the layers, output channel information indicating a number of output channels in each layer, filter kernel size information indicating a size of a filter kernel in each layer, weight information indicating weights of the filter kernel, or bias information indicating a bias for adjusting an output of the filter kernel.

### Claim 5 (depends on 1)

The image decoding method of claim 1, wherein the obtaining the DNN update information comprises:
obtaining default DNN reuse permission information indicating whether a structure of a default DNN is reused for the updated DNN setting information; and
based on the default DNN reuse permission information indicating that the structure of the default DNN is reused for the updated DNN setting information, obtaining reference DNN index information indicating an index of the default DNN of which information is reused, and
wherein the determining the one or more pieces of the updated DNN setting information comprises, based on the default DNN reuse permission information indicating that the structure of the default DNN is reused for the updated DNN setting information, determining the one or more pieces of the updated DNN setting information according to the structure of the default DNN indicated by the reference DNN index information and the DNN update information.

### Claim 6 (depends on 5)

The image decoding method of claim 5, wherein the determining the one or more pieces of the updated DNN setting information further comprises determining a number of layers, a number of input channels in each of the layers, a number of output channels in each layer, and a size of a filter kernel of each layer according to a number of default layers in the default DNN indicated by the reference DNN index information, a number of default input channels in each of the default layers, a number of output channels in each default layer, and a size of a filter kernel of each default layer and determining, from the DNN update information, weights and a bias of the filter kernel in the updated DNN setting information.

### Claim 7 (depends on 6)

The image decoding method of claim 6, wherein the DNN update information comprises at least one of weight information indicating the weights of the filter kernel or bias information indicating the bias for adjusting an output of the filter kernel,
the weight information indicates a difference between weights respectively in the default DNN and an updated DNN, and
the bias information indicates a difference between biases respectively in the default DNN and the updated DNN.

### Claim 8 (depends on 1)

The image decoding method of claim 1, wherein the obtaining the DNN update permission information, the obtaining the DNN update information, and the determining the one or more pieces of the updated DNN setting information are performed for a frame group that is a coded video sequence including a plurality of frames, and
wherein the obtaining the third image comprises:
determining, for a frame in the frame group, the updated DNN setting information used for the frame based on the one or more pieces of the updated DNN setting information; and
obtaining the third image in the frame by performing the AI up-scaling on the second image in the frame according to the updated DNN setting information.

### Claim 9 (depends on 1)

The image decoding method of claim 1, wherein the obtaining the third image comprises:
obtaining, for a frame, DNN index information indicating updated DNN setting information used for the frame based on the one or more pieces of the updated DNN setting information;
determining, according to the DNN index information, the updated DNN setting information used for the frame based on the one or more pieces of the updated DNN setting information;
obtaining secondary DNN update permission information indicating whether a secondary update of the updated DNN setting information is to be performed;
based on the secondary DNN update permission information indicating that the secondary update of the updated DNN setting information is to be performed, obtaining secondary DNN update information; and
obtaining the third image by performing the AI up-scaling on the second image by using secondary updated DNN setting information determined according to the secondary DNN update information.

### Claim 10

An image decoding apparatus comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory to:
obtain deep neural network (DNN) update permission information indicating whether one or more pieces of DNN setting information are updated;
based on the DNN update permission information indicating that the one or more pieces of DNN setting information are updated, obtain DNN update information necessary for determining one or more pieces of the DNN setting information that are updated;
determine the one or more pieces of the updated DNN setting information according to the DNN update information; and
obtain a third image by performing artificial intelligence (AI) up-scaling on a second image according to the one or more pieces of the updated DNN setting information.

### Claim 11

A non-transitory computer-readable recording medium in which artificial intelligence (AI) data is stored, wherein the AI data comprises deep neural network (DNN) update permission information indicating whether one or more pieces of DNN setting information are updated,
based on the DNN update permission information indicating that the one or more pieces of the DNN setting information are updated, the AI data further includes DNN update information necessary for determining one or more pieces of the DNN setting information that are updated, and
DNN index information that indicates whether the updated DNN setting information used for up-scaling a second image based on the one or more pieces of the updated DNN setting information.

### Claim 12 (depends on 11)

The non-transitory computer-readable recording medium of claim 11, wherein the DNN update information comprises DNN number information indicating a number of the one or more pieces of the updated DNN setting information, and
wherein the DNN index information indicates whether the number of the one or more pieces of the updated DNN setting information matches the number of the one or more pieces of the updated DNN setting information indicated by the DNN number information.

### Claim 13 (depends on 11)

The non-transitory computer-readable recording medium of claim 11, wherein the DNN update information further comprises:
default DNN reuse permission information indicating whether a structure of a default DNN is reused for the updated DNN setting information; and
when the default DNN reuse permission information indicates that the structure of the default DNN is reused for the updated DNN setting information, the DNN update information further includes reference DNN index information indicating an index of the default DNN of which information is reused, and
the DNN update information further includes information related to an update of the DNN setting information for which the structure of the default DNN indicated by the reference DNN index information is not reused.

### Claim 14 (depends on 13)

The non-transitory computer-readable recording medium of claim 13, wherein the DNN setting information for which the structure of the default DNN indicated by the reference DNN index information is reused comprises a number of layers in the default DNN, a number of input channels in each of the layers, a number of output channels in each layer, and a size of a filter kernel of each layer,
the DNN setting information for which the structure of the default DNN indicated by the reference DNN index information is not reused comprises weights and a bias of the filter kernel.

### Claim 15 (depends on 11)

The non-transitory computer-readable recording medium of claim 11, wherein the DNN index information is set for each of a plurality of frames in a frame group, and
wherein the AI data further comprises:
secondary DNN update permission information indicating whether a secondary update of the updated DNN setting information determined according to the DNN index information is to be performed, and
when the secondary DNN update permission information indicates that the secondary update of the updated DNN setting information is to be performed, secondary DNN update information necessary for determining secondary updated DNN setting information used for the up-scaling of the second image.
