# US 2022/0284555 A1 · Image processing apparatus and operation method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284555 A1
- Document ID: US20220284555A1
- Status: Published application
- Kind: A1
- Application number: 17/687,343
- Filed: 2022-03-04
- Published: 2022-09-08
- Earliest claimed: 2021-03-04
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Iljun AHN; Hanul SHIN; Jaeyeon PARK; Youngchan SONG; Tammy LEE
- Web page: https://patentgrep.com/patent/US20220284555A1

## CPC

- G06T 5/20: using local operators
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06T 5/30: Erosion or dilatation, e.g. thinning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- G06N 3/048: Activation functions
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20192: Edge enhancement; Edge preservation
- G06T 2207/20224: Image subtraction
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An image processing apparatus, including a memory configured to store one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to obtain similarity information indicating a similarity between each pixel of a plurality of pixels included in a first image and an adjacent pixel of the each pixel; generate a weight map including weight information corresponding to the each pixel, based on the similarity information; generate a spatially variant kernel including a plurality of kernels corresponding to the plurality of pixels, based on the weight map and a spatial kernel including weight information based on a location relationship between the each pixel and the adjacent pixel; and generate a second image by applying the spatially variant kernel to the first image.

## Claims

### Claim 1

An image processing apparatus comprising:
a memory configured to store one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory to:
obtain similarity information indicating a similarity between each pixel of a plurality of pixels included in a first image and an adjacent pixel of the each pixel;
generate a weight map including weight information corresponding to the each pixel, based on the similarity information;
generate a spatially variant kernel including a plurality of kernels corresponding to the plurality of pixels, based on the weight map and a spatial kernel including weight information based on a location relationship between the each pixel and the adjacent pixel; and
generate a second image by applying the spatially variant kernel to the first image.

### Claim 2 (depends on 1)

The image processing apparatus of claim 1, wherein the processor is further configured to execute the one or more instructions to:
obtain first similarity information, based on a difference between the each pixel and a first adjacent pixel at a first relative location with respect to the each pixel; and
obtain second similarity information, based on a difference between the each pixel and a second adjacent pixel at a second relative location with respect to the each pixel.

### Claim 3 (depends on 1)

The image processing apparatus of claim 1, wherein the processor is further configured to execute the one or more instructions to generate the weight map by performing a convolution operation on the similarity information and one or more kernels included in a convolutional neural network, by using the convolutional neural network.

### Claim 4 (depends on 1)

The image processing apparatus of claim 1, wherein a number of channels of at least one of the similarity information, the weight map, or the spatially variant kernel is determined based on a size of the spatial kernel.

### Claim 5 (depends on 1)

The image processing apparatus of claim 1, wherein, in the spatial kernel, a pixel located at a center of the spatial kernel includes a largest value from among values of pixels of the spatial kernel, and
wherein the values of the pixels decrease away from the center of the spatial kernel.

### Claim 6 (depends on 1)

The image processing apparatus of claim 1, wherein a size of the spatial kernel is K×K and a number of channels of the weight map is K2, and
wherein the processor is further configured to execute the one or more instructions to:
list pixel values included in the spatial kernel in a channel direction to convert the pixel values into a weight vector having a size of 1×1×K2; and
generate the spatially variant kernel by multiplying the weight vector by each of 1-dimensional vectors having the size of 1×1×K2 included in the weight map.

### Claim 7 (depends on 1)

The image processing apparatus of claim 1, wherein the spatially variant kernel comprises a number of kernels equal to a number of the plurality of pixels included in the first image.

### Claim 8 (depends on 7)

The image processing apparatus of claim 7, wherein the processor is further configured to execute the one or more instructions to generate the second image by applying a first kernel included in the spatially variant kernel to a first region centered around a first pixel included in the first image and applying a second kernel included in the spatially variant kernel to a second region centered around a second pixel included in the first image.

### Claim 9 (depends on 1)

The image processing apparatus of claim 1, wherein the processor is further configured to execute the one or more instructions to:
generate a first weight map by performing a convolution operation on the similarity information and one or more kernels included in a convolutional neural network, by using the convolutional neural network; and
generate the weight map by performing dilated convolution operation on the first weight map and a first kernel.

### Claim 10 (depends on 9)

The image processing apparatus of claim 9, wherein a number of channels of the first weight map is same as a number of channels of the first kernel, and
wherein the processor is further configured to execute the one or more instructions to generate the weight map by performing a depthwise dilated convolution operation on the first weight map and the first kernel.

### Claim 11

An operation method of an image processing apparatus, the operation method comprising:
obtaining similarity information indicating a similarity between each pixel of a plurality of pixels included in a first image and an adjacent pixel of the each pixel;
generating a weight map including weight information corresponding to the each pixel, based on the similarity information;
generating a spatially variant kernel including a plurality of kernels corresponding to the plurality of pixels, based on the weight map and a spatial kernel including weight information based on a location relationship between the each pixel and the adjacent pixel; and
generating a second image by applying the spatially variant kernel to the first image.

### Claim 12 (depends on 11)

The operation method of claim 11, wherein the obtaining of the similarity information comprises:
obtaining first similarity information, based on a difference between the each pixel and a first adjacent pixel at a first relative location with respect to the each pixel; and
obtaining second similarity information, based on a difference between the each pixel and a second adjacent pixel at a second relative location with respect to the each pixel.

### Claim 13 (depends on 11)

The operation method of claim 11, wherein the weight map is generated by performing a convolution operation on the similarity information and one or more kernels included in a convolutional neural network, by using the convolutional neural network.

### Claim 14 (depends on 11)

The operation method of claim 11, wherein a number of channels of at least one of the similarity information, the weight map, or the spatially variant kernel is determined based on a size of the spatial kernel.

### Claim 15

A non-transitory computer-readable recording medium having stored therein a program which, when executed by at least one processor, causes the at least one processor to:
obtain similarity information indicating a similarity between each pixel of a plurality of pixels included in a first image and an adjacent pixel of the each pixel;
generate a weight map including weight information corresponding to the each pixel, based on the similarity information;
generate a spatially variant kernel including a plurality of kernels corresponding to the plurality of pixels, based on the weight map and a spatial kernel including weight information based on a location relationship between the each pixel and the adjacent pixel; and
generate a second image by applying the spatially variant kernel to the first image.
