# US 2022/0036596 A1 · Method and apparatus for performing artificial intelligence encoding and artificial intelligence decoding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036596 A1
- Document ID: US20220036596A1
- Status: Published application
- Kind: A1
- Application number: 17/503,420
- Filed: 2021-10-18
- Published: 2022-02-03
- Earliest claimed: 2020-06-11
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Chaeeun Lee; Jaehwan Kim; Youngo Park
- Web page: https://patentgrep.com/patent/US20220036596A1

## CPC

- G06T 9/002: {using neural networks}
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06T 3/4046: using neural networks
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- H04N 19/136: Incoming video signal characteristics or properties
- H04N 19/172: the region being a picture, frame or field
- H04N 19/59: involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
- H04N 19/85: using pre-processing or post-processing specially adapted for video compression
- G06N 3/048: Activation functions
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An apparatus for performing artificial intelligence (AI) encoding on an image includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to: determine a resolution of an original image; when the resolution of the original image is higher than a predetermined value, obtain a first image by performing AI downscaling on the original image via a downscaling deep neural network (DNN); when the resolution of the original image is lower than or equal to the predetermined value, obtain a first image by performing AI one-to-one preprocessing on the original image via a one-to-one preprocessing DNN for upscaling; generate image data by performing first encoding on the first image; and transmit the image data and AI data including information related to the AI downscaling or information related to the AI one-to-one preprocessing.

## Claims

### Claim 1

An apparatus for performing artificial intelligence (AI) decoding on an image, the apparatus comprising:
a memory storing one or more instructions; and
at least one processor configured to execute the one or more instructions stored in the memory to:
obtain AI data and image data, the image data generated as a result of first encoding of a first image, and the AI data related to AI downscaling of an original image to the first image or AI one-to-one preprocessing of the original image to the first image;
obtain a second image corresponding to the first image by performing first decoding on the image data;
obtain, based on the AI data, deep neural network (DNN) setting information for AI upscaling of the second image, from among a plurality of pieces of DNN setting information; and
generate a third image by performing the AI upscaling on the second image via an upscaling DNN operating based on the obtained DNN setting information,
wherein the plurality of pieces of DNN setting information are obtained via:
first joint training of the upscaling DNN and a downscaling DNN used for the AI downscaling of the original image, and
second joint training of a one-to-one preprocessing DNN used for the AI one-to-one preprocessing of the original image and the upscaling DNN, the second joint training being performed using DNN setting information for the AI upscaling obtained as a result of the first joint training.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the one-to-one preprocessing DNN is configured to enhance features of the original image while maintaining a same resolution as that of the original image.

### Claim 3

A method for performing artificial intelligence (AI) decoding on an image, the method comprising:
obtaining AI data and image data, the image data generated as a result of first encoding of a first image, and the AI data related to AI downscaling of an original image to the first image or AI one-to-one preprocessing of the original image to the first image;
obtaining a second image corresponding to the first image by performing first decoding on the image data;
obtaining, based on the AI data, deep neural network (DNN) setting information for AI upscaling of the second image, from among a plurality of pieces of DNN setting information; and
generating a third image by performing the AI upscaling on the second image via an upscaling DNN operating based on the obtained DNN setting information,
wherein the plurality of pieces of DNN setting information are obtained via:
first joint training of the upscaling DNN and a downscaling DNN used for the AI downscaling of the original image, and
second joint training of a one-to-one preprocessing DNN used for the AI one-to-one preprocessing of the original image and the upscaling DNN, the second joint training being performed using DNN setting information for the AI upscaling obtained as a result of the first joint training.

### Claim 4 (depends on 3)

The method of claim 3, wherein the one-to-one preprocessing DNN is configured to enhance features of the original image while maintaining a same resolution as that of the original image.

### Claim 5 (depends on 3)

A non-transitory computer-readable recording medium having recorded thereon a program for performing the method of claim 3.