# US 2021/0334642 A1 · Apparatus and method for image processing, and system for training neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0334642 A1
- Document ID: US20210334642A1
- Status: Published application
- Kind: A1
- Application number: 16/604,410
- Filed: 2019-04-23
- Published: 2021-10-28
- Earliest claimed: 2018-09-30
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: Boe Technology Group Co., Ltd.
- Inventors: Pablo Navarrete Michelini; Dan Zhu; Hanwen Liu
- Web page: https://patentgrep.com/patent/US20210334642A1

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 3/4046: using neural networks
- G06T 3/4076: using the original low-resolution images to iteratively correct the high-resolution images
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 5/90: Dynamic range modification of images or parts thereof
- G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/82: using neural networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20212: Image combination
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure generally relates to the field of deep learning technologies. An apparatus for generating a plurality of correlation images may include a feature extracting unit configured to receive a training image and extracting at least one or more of feature from the training image to generate a first feature image based on the training image; a normalizer configured to normalize the first feature image and generate a second feature image; and a shift correlating unit configured to perform a plurality of translational shifts on the second feature image to generate a plurality of shifted images, correlate each of the plurality of shifted images with the second feature image to generate the plurality of correlation images.

## Claims

### Claim 1

An apparatus for generating a plurality of correlation images, comprising:
a feature extracting unit configured to receive a training image and extracting at least one or more of feature from the training image to generate a first feature image based on the training image;
a normalizer configured to normalize the first feature image and generate a second feature image; and
a shift correlating unit configured to perform a plurality of translational shifts on the second feature image to generate a plurality of shifted images, correlate each of the plurality of shifted images with the second feature image to generate the plurality of correlation images.

### Claim 2 (depends on 1)

The apparatus according to claim 1,
wherein the shift correlating unit is configured to perform the plurality of translational shifts on the second feature image by shifting a number of leftmost or rightmost columns of pixels in the pixel block of the second feature image to be the rightmost and leftmost column, respectively, of the pixel block; and shifting b number of bottommost or topmost rows of pixels in the pixel block of the second feature image to be the topmost or bottommost row, respectively, of the pixel block,
wherein 0≤a<Y, 0≤b<X, each of a and b being an integer, Y being a total number of columns of pixels in the pixel block of the second feature image, and X being a total number of rows of pixels in the pixel block of the second feature image, and
wherein a and b may be the same or different.

### Claim 3 (depends on 1)

The apparatus according to claim 1,
wherein the shift correlating unit is configured to perform the plurality of translational shifts on the second feature image by shifting a number of leftmost or rightmost columns of pixels in the pixel block of the second feature image to be the rightmost and leftmost column, respectively, of the pixel block; and shifting b number of bottommost or topmost rows of pixels in the pixel block of the second feature image to be the topmost or bottommost row, respectively, of the pixel block,
wherein 0≤a<Y, 0≤b<X, each of a and b being an integer, Y being a total number of columns of pixels in the pixel block of the second feature image, and X being a total number of rows of pixels in the pixel block of the second feature image, and
wherein a and b may be the same or different.

### Claim 4 (depends on 1)

The apparatus according to claim 1,
wherein the shift correlating unit is configured to correlate each of the plurality of shifted images with the second feature image by multiplying a pixel value of each pixel in the pixel block of each of the plurality of shifted images with a pixel value of a positionally corresponding pixel in the pixel block of the second feature image.

### Claim 5 (depends on 1)

The apparatus according to claim 1,
wherein the first feature image is a luminance feature image, and
wherein the feature extracting unit comprises:
a luminance detector configured to extract luminance information from the training image to generate the luminance feature image.

### Claim 6 (depends on 5)

The apparatus according to claim 5,
wherein to generate the luminance feature image, the luminance detector is configured to determine a luminance value of a pixel at a given position in the luminance feature image according to the following formula (1):
I=0.299R+0.587G+0.114B (1)
wherein:
I is the luminance value,
R is a red component value of a positionally corresponding pixel in the training image,
G is a green component value of the positionally corresponding pixel in the training image, and
B is a blue component value of the positionally corresponding pixel in the training image.

### Claim 7 (depends on 5)

The apparatus according to claim 5,
wherein the normalizer is configured to normalize the luminance feature image according to the following formula (2):
{
N
=
I
-
μ
σ
-
1
σ
2
=
Blur
⁢
⁢
(
I
2
)
-
μ
μ
=
⁢
Blur
⁢
⁢
(
I
)
(
2
)
wherein:
N is the first feature image,
I represents the luminance value of a pixel at a given position in the luminance feature image,
Blur(I) is an image obtained by applying a Gaussian filter to the luminance feature image, and
Blur(I2) is an image obtained by squaring every pixel value in the luminance feature image, and then applying the Gaussian filter to the image.

### Claim 8 (depends on 1)

The apparatus according to claim 1,
wherein the second feature image comprises a pixel block having a first size,
wherein each of the plurality of shifted images and each of the plurality of correlation images comprises a pixel block having the first size, and
wherein in each of the plurality of shifted images, a pixel having a non-zero pixel value has a corresponding pixel with the same non-zero pixel value in the second feature image.

### Claim 9

A method of generating a plurality of correlation images, the method comprising:
generating a first feature image based on a training image;
normalizing the first feature image and generating a second feature image;
performing a plurality of translational shifts on the second feature image to generate a plurality of shifted images; and
correlating each of the plurality of shifted images with the second feature image to generate a plurality of correlation images.

### Claim 10 (depends on 9)

The method according to claim 9,
wherein the correlating of each of the plurality of shifted images with the second feature image comprises multiplying a pixel value of each pixel in the pixel block of each of the plurality of shifted images with a pixel value of a positionally corresponding pixel in the pixel block of the second feature image.

### Claim 11 (depends on 9)

The method according to claim 9,
wherein the performing of the plurality of translational shifts comprises:
shifting a number of leftmost or rightmost columns of pixels in the pixel block of the second feature image to be the rightmost and leftmost column, respectively, of the pixel block; and
shifting b number of bottommost or topmost rows of pixels in the pixel block of the second feature image to be the topmost or bottommost row, respectively, of the pixel block,
wherein 0≤a<Y, 0≤b<X, each of a and b being an integer, Y being a total number of columns of pixels in the pixel block of the second feature image, and X being a total number of rows of pixels in the pixel block of the second feature image, and
wherein a and b may be the same or different.

### Claim 12 (depends on 11)

The method according to claim 11,
wherein at least one of a and b changes at least once during the performing of the plurality of translational shifts.

### Claim 13 (depends on 9)

The method according to claim 9,
wherein the performing of the plurality of translational shifts comprises:
deleting a number of leftmost or rightmost columns of pixels in the pixel block of the second feature image, and adding a number of columns of pixels at the rightmost or leftmost position, respectively, of the pixel block; and
deleting b number of bottommost or topmost rows of pixels in the pixel block of the second feature image, and adding b number of rows at the topmost or bottommost position, respectively, of the pixel block,
wherein 0≤a<Y, 0≤b<X, each of a and b being an integer, Y being a total number of columns of pixels in the pixel block of the second feature image, and X being a total number of rows of pixels in the pixel block of the second feature image, and
wherein each of the added pixels has a pixel value of 0.

### Claim 14 (depends on 13)

The method according to claim 13,
wherein at least one of a and b changes at least once during the performing of the plurality of translational shifts.

### Claim 15 (depends on 9)

The method according to claim 9, further comprising performing X*Y translational shifts, Y being a total number of columns of pixels in the pixel block of the second feature image, and X being a total number of rows of pixels in the pixel block of the second feature image.

### Claim 16 (depends on 9)

The method according to claim 9, further comprising, before the generating of the first feature image, receiving the training image,
wherein the generating of the first feature image comprises generating a luminance feature image based on luminance information of the training image.

### Claim 17 (depends on 16)

The method according to claim 16, further comprising determining a luminance value of a pixel at a given position in the luminance feature image according to the following formula (1):
I=0.299R+0.587G+0.114B (1)
wherein:
I is the luminance value,
R is a red component value of a positionally corresponding pixel in the training image,
G is a green component value of the positionally corresponding pixel in the training image, and
B is a blue component value of the positionally corresponding pixel in the training image.

### Claim 18 (depends on 16)

The method according to claim 16, further comprising normalizing the luminance feature image according to the following formula (2):
{
N
=
I
-
μ
σ
-
1
σ
2
=
Blur
⁢
⁢
(
I
2
)
-
μ
μ
=
⁢
Blur
⁢
⁢
(
I
)
(
2
)
wherein:
N is the first feature image,
I represents the luminance feature image,
Blur(I) is an image obtained by applying a Gaussian filter to the luminance feature image, and
Blur(I2) is an image obtained by squaring every pixel value in the luminance feature image, and then applying the Gaussian filter to the image.

### Claim 19 (depends on 9)

The method according to claim 9,
wherein the first feature image comprises a pixel block having a first size,
wherein each of the plurality of shifted images and each of the plurality of correlation images comprises a pixel block having the first size, and
wherein in each of the plurality of shifted images, a pixel having a non-zero pixel value has a corresponding pixel with the same non-zero pixel value in the first feature image.

### Claim 20 (depends on 9)

A non-transitory computer-readable medium storing instructions that cause a computer to execute the method according to claim 9.

### Claim 21 (depends on 1)

A system for training a generative adversarial network, comprising
a generative adversarial network processor, comprising a generative network microprocessor which is configured to be trained by a discriminative network microprocessor, and the discriminative network microprocessor which is coupled to the generative network,
wherein the discriminative network microprocessor comprises:
a plurality of input terminals coupled to a plurality of the apparatuses according to claim 1;
a plurality of analysis modules, each of the plurality of analysis modules being coupled to one of the plurality of input terminals;
a plurality of pooling modules connected in cascade, each stage of the cascade comprising a pooling module coupled to one of the plurality of analysis modules and to a pooling module in a previous stage of the cascade; and
a discriminator network coupled to the pooling module in a last stage of the cascade.