# US 2021/0342976 A1 · Apparatus, method, and computer-readable medium for image processing, and system for training a neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0342976 A1
- Document ID: US20210342976A1
- Status: Published application
- Kind: A1
- Application number: 16/614,558
- Filed: 2019-06-20
- Published: 2021-11-04
- Earliest claimed: 2018-09-30
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: Boe Technology Group Co., Ltd.
- Inventors: Pablo Navarrete Michelini; Dan Zhu; Hanwen Liu
- Web page: https://patentgrep.com/patent/US20210342976A1

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/241: relating to the classification model, e.g. parametric or non-parametric approaches
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 3/20: Linear translation of whole images or parts thereof, e.g. panning
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 3/4046: using neural networks
- G06T 3/4076: using the original low-resolution images to iteratively correct the high-resolution images
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/70: Denoising; Smoothing
- G06T 5/90: Dynamic range modification of images or parts thereof
- G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/82: using neural networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20212: Image combination
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure provides a method of training a generative adversarial network. The method includes iteratively enhancing a first noise input in a generative network to generate a first output image; iteratively enhancing a second noise input in the generative network to generate a second output image; transmitting the first output image and a second reference image to a discriminative network, the second reference image corresponding to the first reference image and having a higher resolution than the first reference image; obtaining a first score from the discriminative network based on the second reference image, and a second score from the discriminative network based on the first output image; calculating a loss function of the generative network based on the first score and the second score; and adjusting at least one parameter of the generative network to lower the loss function of the generative network.

## Claims

### Claim 1

A method of training a generative adversarial network, the method comprising:
iteratively enhancing a first noise input having a first amplitude and a first reference image in a generative network to generate a first output image,
iteratively enhancing a second noise input having a second amplitude and the first reference image in the generative network to generate a second output image,
transmitting the first output image and a second reference image to a discriminative network, the second reference image corresponding to the first reference image and having a higher resolution than the first reference image,
obtaining a first score from the discriminative network based on the second reference image, and a second score from the discriminative network based on the first output image,
calculating a loss function of the generative network based on the first score and the second score, and
adjusting at least one parameter of the generative network to lower the loss function of the generative network.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the loss function of the generative network is calculated according to equation (1):
Loss=λ1Lrec(X, Yn=0)+λ2Lper(X, Yn=1)+λ3LGAN(Yn=1) (1),
X represents the high-resolution reference image,
Yn=0 represents the second output image,
Yn=1 represents the first output image,
Lrec(X, Yn=0) represents a reconstruction error between the second output image and the second reference image,
Lper(X, Yn=1) represents a perceptual loss between the first output image and the second reference image,
LGAN(Yn=1) represents a sum of the first score and the second score, and
λ1, λ2, λ3 each presents a predetermined weight.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the reconstruction error between the second output image and the second reference image is calculated according to equation (2):
Lrec(X, Yn=0)=E[Σl=1L∥Yn=0l−HRl∥1]+E[Σl=1L∥Dbicl(Yn=0l)−LR∥1] (2).
L represents the number of iterations of enhancement, and L≥1.
Yn-0l represents an image generated by the generative network microprocessor after performing one iteration based on the second noise input, and l≤L,
LR represents first reference image,
Dbixl(Yn=0l) represents an image obtained by performing downsampling on the image represented by Yn=0l, the image represented by Dbicl(Yn=0l) having the same resolution has the first reference image,
HRl represents an image obtained by performing downsampling on the second reference image, the image represented by HRl having the same resolution as the image represented by Yn=0l, and
E[ ] represents a matrix energy calculation.

### Claim 4 (depends on 2)

The method according to claim 2, wherein the perceptual loss between the first output image and the second reference image is calculated according to equation (3):
Lper(X, Yn=1)=e[Σl=1LLCX(Yn=1l, HRl)]+E[Σl=1LLCX(Dbixl(Yn=1l), LR)] (3),
Yn=1l represents an image generated by the generative network microprocessor after performing one iteration based on the first noise input,
Dbicl(Yn=1l) represents an image obtained by performing downsampling on the image represented by Yn=1l, the image represented by Dbicl(Yn=1l) having the same resolution as the first reference image, and
LCX( ) represents a perceptual loss function.

### Claim 5 (depends on 2)

The method according to claim 2, wherein the sum of the first score and the second score is calculated according to equation (4):
LGAN(Yn=1)=E[log(D(HR))]+E[log (1−D(Yn=1))] (4).
D(Yn=1) represents the first score, and
D(HR) represents the second score.

### Claim 6 (depends on 1)

The method according to claim 1, further comprising:
providing the first noise input and the first reference image to the generative network to generate a third output image, the at least one parameter of the generative network having been adjusted,
providing the third output image and the second reference image to discriminative network,
obtaining a third score from the discriminative network based on the second reference image, and a fourth score from the discriminative network based on the third output image, and
calculating the loss function of the generative network microprocessor.

### Claim 7 (depends on 1)

The method according to claim 1, wherein the iteratively enhancing of the first noise input and the first reference image comprises:
generating a first feature image based on the first reference image,
concatenating the first feature image with the first noise input to obtain a first composite image, and
iteratively enhancing, in a finite number of iterations, the first reference image based on the first feature image and the first composite image to generate a higher-resolution image of the first reference image,
wherein the noise input for each of the finite number of iterations has the same predetermined amplitude.

### Claim 8 (depends on 7)

The method according to claim 7, further comprising:
interpolating the first reference image to obtain a first interpolated image,
generating a second feature image based on the first interpolated image,
downsampling the second feature image, and joining the downsampled second feature image with the first composite image to obtain a second composite image, and
iteratively enhancing, in the finite number of iterations, the first reference image based on the second feature image, the first composite image, and the second composite image to obtain the higher-resolution image of the first reference image.

### Claim 9 (depends on 7)

The method according to claim 7, further comprising:
generating a first residue image based on the first composite image, the first residue image representing a degree of dissimilarity between the first composite image and the first feature image, and
applying a residual correction to the first feature image based on the first residue image to obtain the higher-resolution image of the first reference image.

### Claim 10 (depends on 9)

The method according to claim 9, wherein the generating of the first residue image and the applying of the residual correction are performed at least once.

### Claim 11 (depends on 1)

A system for training a generative adversarial network, comprising:
a generative adversarial network processor, comprising a generative network microprocessor, and a discriminative network microprocessor coupled to the generative network microprocessor,
wherein the generative adversarial network processor is configured to perform the method according to claim 1.

### Claim 12 (depends on 1)

A system, comprising a generative network microprocessor that has been trained by the method according to claim 1.

### Claim 13 (depends on 7)

A generative network microprocessor, comprising an apparatus configured to perform the method according to claim 7.

### Claim 14 (depends on 13)

The generative network microprocessor according to claim 13, wherein the apparatus comprises:
an analysis processor,
a connection processor coupled to the analysis processor, and
an enhancement processor coupled to the connection processor,
wherein:
the analysis processor is configured to receive a reference image, and extract one or more feature from the input image to generate a feature image based on the reference image,
the connection processor is configured to receive a noise input having a predetermined amplitude, and concatenate the noise input and the feature image to generate a first composite image,
the enhancement processor is configured to iteratively enhance the reference image based on the feature image and the first composite image to generate a higher-resolution image of the reference image,
when a plurality of iterations are performed, the noise input for each iteration has the same predetermined amplitude.

### Claim 15 (depends on 14)

The generative network microprocessor according to claim 14, wherein:
the enhancement processor comprises a first upsampler, a downsampler, a residue determination processor, a second upsampler, a correction processor, and a synthesis processor, which are coupled to each other,
the first upsampler is configured to upsample the first composite image to generate an upsampled feature image,
the downsampler is configured to downsample the unsampled feature image to generate a downsampled feature image,
the residue determination processor is configured to generate a residue image from the downsampled feature image and the first composite image, the residue image representing a difference between the downsampled feature image and the first composite image,
the second upsampler is configured to upsample the residue image to generate a upsampled residue image,
the correction processor is configured to apply at least one residual correction to the upsampled feature image according to the upsampled residue image to generate a higher-resolution feature image of the reference image,
the synthesis processor is configured to synthesize a higher-resolution image of the reference image from the higher-resolution feature image,
the enhancement processor is configured to perform at least two iterations, and
the higher-resolution image and the higher-resolution feature image are a reference image and a feature image for a subsequent iteration.

### Claim 16 (depends on 15)

The generative network microprocessor according to claim 15, wherein:
the enhancement processor further comprises an interpolation processor and a superposition processor, which are coupled to each other,
the interpolation processor is configured to perform interpolation on the reference image to generate an interpolated image, and
the superposition processor is configured to superpose the interpolated image on an output from the synthesis processor to generate the higher-resolution image of the reference image.

### Claim 17 (depends on 15)

The generative network microprocessor according to claim 15, wherein:
the first upsampler is configured to upsample the first composite image directly.

### Claim 18 (depends on 17)

The generative network microprocessor according to claim 17, wherein: the first upsampler is configured to generate a second composite image based on the interpolated image, and then upsample the second composite image to generate the upsampled feature image.
