# US 2021/0342973 A1 · Image upscaling apparatus using artificial neural network having multiple deconvolution layers and deconvolution layer pluralization method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342973 A1
- Document ID: US20210342973A1
- Status: Published application
- Kind: A1
- Application number: 17/029,478
- Filed: 2020-09-23
- Published: 2021-11-04
- Earliest claimed: 2020-04-29
- Assignees: Industry-Academic Cooperation Foundation, Yonsei University
- Applicants: Industry-Academic Cooperation Foundation, Yonsei University
- Inventors: Seong Ook JUNG; Sung Hwan JOO; Su Min LEE
- Web page: https://patentgrep.com/patent/US20210342973A1

## CPC

- G06T 3/4046: using neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 5/20: using local operators
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Disclosed is an image upscaling apparatus that includes: multiple convolution layers, each configured to receive an input image or a feature map outputted by a previous convolution layer and extract features to output a feature map; and a multilayer configured to receive a final feature map outputted from the last convolution layer and output an upscaled output image. The multilayer includes: a first partition layer including first filters having a minimum size along the x-axis and y-axis directions and the same size as the final feature map along the z-axis direction; and at least one second partition layer, each including second filters, having a size greater than that of the first filter in the x-axis and y-axis directions and having a number and size of the first filter in the z-axis direction, and configured to shuffle features in the x-axis and y-axis directions of the first shuffle map.

## Claims

### Claim 1

An image upscaling apparatus comprising:
a plurality of convolution layers each configured to receive an input image or a feature map outputted by a previous convolution layer and output a feature map by extracting features from the input image or from the previous feature map; and
a multilayer configured to receive a final feature map outputted from a last convolution layer among the plurality of convolution layers and output an output image upscaled to have a predefined resolution higher than a resolution of the input image,
wherein the multilayer comprises:
a first partition layer including a plurality of first filters and configured to output a first shuffle map, the first filters having a minimum size along an x-axis and a y-axis direction and having a same size as a z-axis direction size of the final feature map along a z-axis direction, the x-axis, y-axis, and z-axis directions representing width, height, and depth directions of the final feature map; and
at least one second partition layer each including a plurality of second filters and configured to shuffle features in the x-axis and y-axis directions of the first shuffle map, the second filters having a size greater than that of the first filter in the x-axis and y-axis directions and having a same size of the first filter in the z-axis direction.

### Claim 2 (depends on 1)

The image upscaling apparatus of claim 1, wherein the first partition layer outputs the first shuffle map by having the plurality of first filters perform MAC (multiply-accumulate) operations while moving in units of elements along the x-axis and y-axis directions in the final feature map and arranging computed values obtained as results of the MAC operations at positions corresponding to x-axis and y-axis positions of operand elements but with the computed values arranged at different predefined positions along the z-axis direction in correspondence with the plurality of first filters, respectively.

### Claim 3 (depends on 2)

The image upscaling apparatus of claim 2, wherein a number of first filters included in the first partition layer corresponds to a ratio of increase in resolution between the input image and the output image.

### Claim 4 (depends on 3)

The image upscaling apparatus of claim 3, wherein each of the at least one second partition layers includes a same number of second filters as a number of the first filters of the first partition layer.

### Claim 5 (depends on 4)

The image upscaling apparatus of claim 4, wherein the at least one second partition layer outputs a second shuffle map by having the plurality of second filters perform MAC operations while moving in units of elements along the x-axis and y-axis directions in the first shuffle map or a second shuffle map received from a previous second partition layer and arrange computed values obtained as results of the MAC operations at positions corresponding to x-axis and y-axis positions of operand elements of the received first shuffle map or second shuffle map but with the computed values arranged at different predefined positions along the z-axis direction in correspondence with the plurality of second filters, respectively.

### Claim 6 (depends on 5)

The image upscaling apparatus of claim 5, wherein a number of the second partition layers and a size of the second filters are set such that a total number of computations required of the multilayer including the first partition layer and the at least one second partition layer does not exceed a predefined threshold.

### Claim 7 (depends on 1)

The image upscaling apparatus of claim 1, wherein each of the plurality of convolution layers comprises at least one convolution filter having a predefined size and having weights obtained by learning mapped therein, the convolution layer configured to receive the input image or a feature map outputted from a previous convolution layer, the convolution filter configured to extract features of the received input image or feature map to output a feature map.

### Claim 8 (depends on 7)

The image upscaling apparatus of claim 7, wherein the final feature map has an x-axis direction size and a y-axis direction size corresponding to the resolution of the input image and a z-axis direction size corresponding to a number of convolution filters included in a last convolution layer among the plurality of convolution layers.

### Claim 9 (depends on 1)

The image upscaling apparatus of claim 1, wherein the multilayer further comprises a remapping layer configured to expand and rearrange elements arranged in the z-axis direction in a second shuffle map such that the rearranged elements are placed adjacent to each other in a 2-dimensional plane according to a predefined pattern and arrangement order, the second shuffle map outputted from a last second partition layer among the at least one second partition layers.

### Claim 10

A deconvolution layer pluralization method for an image upscaling apparatus, the deconvolution layer pluralization method comprising:
setting a plurality of convolution layers, the plurality of convolution layers configured to receive an input image or a feature map outputted by a previous convolution layer and output a feature map by extracting features from the input image or from the previous feature map; and
setting a multilayer as a deconvolution layer configured to receive a final feature map outputted from a last convolution layer among the plurality of convolution layers and output an output image upscaled to have a predefined resolution higher than a resolution of the input image,
wherein the setting of the multilayer comprises:
setting a first partition layer, the first partition layer including a plurality of first filters and configured to output a first shuffle map, the first filters having a minimum size along an x-axis and a y-axis direction and having a same size as a z-axis direction size of the final feature map along a z-axis direction, the x-axis, y-axis, and z-axis directions representing width, height, and depth directions of the final feature map; and
setting at least one second partition layer, the at least one second partition layer each including a plurality of second filters and configured to shuffle features in the x-axis and y-axis directions of the first shuffle map, the second filters having a size greater than that of the first filter in the x-axis and y-axis directions and having a number and size of the first filter in the z-axis direction.

### Claim 11 (depends on 10)

The deconvolution layer pluralization method of claim 10, wherein the setting of the first partition layer comprises:
setting the plurality of first filters to perform MAC (multiply-accumulate) operations while moving in units of elements along the x-axis and y-axis directions in the final feature map; and
configuring the first partition layer to output the first shuffle map by arranging computed values obtained as results of the MAC operations at positions corresponding to x-axis and y-axis positions of operand elements but with the computed values arranged at different predefined positions along the z-axis direction in correspondence with the plurality of first filters, respectively.

### Claim 12 (depends on 11)

The deconvolution layer pluralization method of claim 11, wherein the setting of the plurality of first filters comprises setting the number of first filters in correspondence to a ratio of increase in resolution between the input image and the output image.

### Claim 13 (depends on 12)

The deconvolution layer pluralization method of claim 12, wherein the setting of the at least one second partition layers comprises setting a number of second filters included in each of the at least one second partition layers to be equal to the number of first filters in the first partition layer.

### Claim 14 (depends on 13)

The deconvolution layer pluralization method of claim 13, wherein the setting of the at least one second partition layers comprises:
setting the plurality of second filters to perform MAC operations while moving in units of elements along the x-axis and y-axis directions in the first shuffle map or a second shuffle map received from a previous second partition layer; and
configuring the at least one second partition layers to output a second shuffle map by arranging computed values obtained as results of the MAC operations at positions corresponding to x-axis and y-axis positions of operand elements of the received first shuffle map or second shuffle map but with the computed values arranged at different predefined positions along the z-axis direction in correspondence with the plurality of second filters, respectively.

### Claim 15 (depends on 14)

The deconvolution layer pluralization method of claim 14, wherein the setting of the at least one second partition layers comprises setting a number of the second partition layers and a size of the second filters such that a total number of computations required of the multilayer including the first partition layer and the at least one second partition layer does not exceed a predefined threshold.

### Claim 16 (depends on 10)

The deconvolution layer pluralization method of claim 10, wherein the setting of the plurality of convolution layers comprises:
setting a number of the convolution layers by receiving the input image or a feature map outputted from a previous convolution layer, and extracting features of the received input image or feature map to output a feature map; and
setting the number and size of convolution filters to which weights obtained by learning are mapped with a predetermined size for each of the set convolution layers.

### Claim 17 (depends on 16)

The deconvolution layer pluralization method of claim 16, wherein the final feature map has an x-axis direction size and a y-axis direction size corresponding to the resolution of the input image and a z-axis direction size corresponding to a number of convolution filters included in a last convolution layer among the plurality of convolution layers.

### Claim 18 (depends on 10)

The deconvolution layer pluralization method of claim 10, wherein the setting of the multilayer comprises setting a remapping layer to expand and rearrange elements arranged in the z-axis direction in a second shuffle map such that the rearranged elements are placed adjacent to each other in a 2-dimensional plane according to a predefined pattern and arrangement order, the second shuffle map outputted from a last second partition layer among the at least one second partition layers.
