# US 10,769,758 B2 · Resolving method and system based on deep learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,769,758 B2
- Document ID: US10769758B2
- Status: Granted
- Kind: B2
- Application number: 15/537,677
- Filed: 2016-06-21
- Published: 2018-03-29
- Granted: 2020-09-08
- Earliest claimed: 2016-03-21
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: BOE Technology Group Co., Ltd.
- Inventors: Lijie Zhang; Zhenglong Li
- Web page: https://patentgrep.com/patent/US10769758B2

## CPC

- G06V 10/82: using neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 3/4076: using the original low-resolution images to iteratively correct the high-resolution images
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 40/162: {using pixel segmentation or colour matching}
- G06V 40/168: {Feature extraction; Face representation}
- G06N 3/02: Neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A resolving method and a system, the method includes: creating a sample library by utilization of an original high-resolution (HR) image set; training a convolutional structural network by utilization of the sample library; and obtaining an HR output signal by processing a low-resolution (LR) input signal by utilization of the trained convolutional structural network. In the resolving method and system according to this disclosure, data after expansion may be processed by simple expansion hardware, without a large change algorithm; and complex algorithms are allocated into parallelizing design, and different servers operate mutually independent; and also, due to the modular design, the design proposals of functional modules may be modified by latter optimization.

## Claims

### Claim 1

A resolving method, comprising:
creating a sample library by utilization of an original high-resolution (HR) image set;
training a convolutional structural network by utilization of the sample library; and
obtaining an HR output signal by processing a low-resolution (LR) input signal by utilization of the trained convolutional structural network,
wherein the convolutional structural network is formed by an alternate connection of a plurality of convolutional layers and excitation layers, and
each convolutional layer includes a plurality of filter units with adjustable filtering parameters,
wherein the sample library includes a face feature sample library, and
the creating the sample library by utilization of the original HR image set further includes:
obtaining an LR image set by a downsampling of the original HR image set;
extracting face feature information of LR images by a face feature extraction method;
obtaining face feature information of HR images by marking face feature points on the HR images; and
creating the face feature sample library, including pairs of the face feature information of the LR images and relevant face feature information of the HR images, by utilization of the face feature information of the LR images and the face feature information of the HR images,
wherein the training the convolutional structural network by utilization of the sample library further includes:
obtaining a high-pass filtering face result and a low-pass filtering face result by respectively performing high-pass filtering and low-pass filtering on the face feature information of the HR images; and obtaining a detail template of the face feature information of the HR images as a feedback signal of the convolutional structural network by a superimposition and a feature classification of the high-pass filtering face result and the low-pass filtering face result, wherein the high-pass filtering face result including structure and contour information of the face, the low-pass filtering face result including skin texture and roughness of the face.

### Claim 2 (depends on 1)

The resolving method according to claim 1, wherein the training the convolutional structural network by utilization of the sample library further includes:
obtaining first filtering parameters for the convolutional structural network by analyzing a correlation between the pairs of the face feature information of the LR images and the relevant face feature information of the HR images in the face feature sample library;
and
adopting the face feature information of the LR images as an input signal of the convolutional structural network, adjusting the first filtering parameters in the convolutional structural network, and obtaining a forecast result signal, being the same with the feedback signal, by processing the input signal by utilization of the convolutional structural network according to the adjusted first filtering parameters.

### Claim 3 (depends on 2)

The resolving method according to claim 2, wherein the obtaining the HR output signal by processing the LR input signal by utilization of the trained convolutional structural network further includes:
inputting the face feature information of the LR images;
processing the inputted face feature information of the LR images by utilization of the convolutional structural network according to the adjusted first filtering parameters; and
outputting face feature information of the HR images, processed by the convolutional structural network.

### Claim 4 (depends on 2)

The resolving method according to claim 2, wherein the sample library includes an image sample library, and the creating the sample library by utilization of the original HR image set further includes:
creating the image sample library, including pairs of the LR images and relevant HR images, by utilization of the LR image set and the HR image set.

### Claim 5 (depends on 4)

The resolving method according to claim 4, wherein the training the convolutional structural network by utilization of the sample library further includes:
obtaining second filtering parameters of the convolutional structural network by analyzing a correlation between the pairs of the LR images and the relevant HR images;
obtaining a high-pass filtering result and a low-pass filtering result by respectively performing high-pass filtering and low-pass filtering on the HR images; obtaining a detail template of the HR images as a feedback signal of the convolutional structural network by a superimposition and a feature classification of the high-pass filtering result and the low-pass filtering result; and
adopting the LR images as an input signal of the convolutional structural network, adjusting the second filtering parameters in the convolutional structural network, and obtaining a forecast result signal, being the same with the feedback signal, by processing the input signal by utilization of the convolutional structural network according to the adjusted second filtering parameters.

### Claim 6 (depends on 5)

The resolving method according to claim 5, wherein the obtaining the HR output signal by processing the LR input signal by utilization of the trained convolutional structural network further includes:
inputting the LR images;
processing the inputted LR images by utilization of the convolutional structural network according to the adjusted second filtering parameters; and
outputting HR images processed by the convolutional structural network.

### Claim 7 (depends on 6)

The resolving method according to claim 6, wherein each filter unit adopts a formula F(x)=Wx+b to execute a convolution operation, where W and b refer to the filtering parameters; x refers to input; and F(x) refers to output.

### Claim 8 (depends on 7)

The resolving method according to claim 7, wherein the forecast result signal is determined to be the same with the feedback signal when J(W,b) obtained according to the following formula is less than a first threshold:
J
⁡
(
W
,
b
)
=
1
m
⁢
∑
i
=
1
m
⁢
(
1
2
⁢

h
W
,
b
⁡
(
I
HR
i
-
I
HR
_
i
)

2
)
where J(W,b) refers to a mean square error; m refers to a number of image sets in the face feature sample library; IHRi refers to the feedback signal; IHRi refers to the forecast result signal; and hW,b refers to a weight coefficient.

### Claim 9 (depends on 8)

The resolving method according to claim 8, wherein when the forecast result signal is different from the feedback signal, a partial derivative of J(W,b) is calculated for each filtering parameter, and the first filtering parameters or the second filtering parameters are adjusted according to the partial derivatives.

### Claim 10 (depends on 9)

The resolving method according to claim 9, wherein the first filtering parameters are classifier filtering parameters for the convolutional structural network.

### Claim 11

A resolving system, comprising:
a sample library creating device configured to create a sample library by utilization of an original high-resolution (HR) image set;
a training device configured to train a convolutional structural network by utilization of the sample library; and
an output device configured to obtain an HR output signal by processing a low-resolution (LR) input signal by utilization of the trained convolutional structural network,
wherein the convolutional structural network is formed by an alternate connection of a plurality of convolutional layers and excitation layers, and
each convolutional layer includes a plurality of filter units with adjustable filtering parameters,
wherein the sample library includes a face feature sample library, and
the sample library creating device further includes:
a downsampling unit configured to obtain an LR image set by a downsampling of the original HR image set;
a face analysis unit configured to extract face feature information of LR images from the LR images by a face feature extraction method;
a feature point marking unit configured to obtain face feature information of HR images by marking face feature points on the HR images; and
a face feature sample library creating unit configured to create a face feature sample library, including pairs of the face feature information of the LR images and relevant face feature information of the HR images, by utilization of the face feature information of the LR images and the face feature information of the HR images,
wherein the training device further includes a first training unit configured to obtain a high-pass filtering face result and a low-pass filtering face result by respectively performing high-pass filtering and low-pass filtering on the face feature information of the HR images; and obtain a detail template of the face feature information of the HR images as a feedback signal of the convolutional structural network by a superimposition and a feature classification of the high-pass filtering face result and the low-pass filtering face result, wherein the high-pass filtering face result including structure and contour information of the face, the low-pass filtering face result including skin texture and roughness of the face.

### Claim 12 (depends on 11)

The resolving system according to claim 11, wherein the training device further includes:
a first training unit configured to: obtain first filtering parameters of the convolutional structural network by analyzing a correlation between the pairs of the face feature information of the LR images and the relevant face feature information of the HR images in the face feature sample library; and adopt the face feature information of the LR images as an input signal of the convolutional structural network, adjust the first filtering parameters in the convolutional structural network, and obtain a forecast result signal, being the same with the feedback signal, by processing the input signal by utilization of the convolutional structural network according to the adjusted first filtering parameters.

### Claim 13 (depends on 12)

The resolving system according to claim 12, wherein the sample library includes an image sample library, and the sample library creating device further includes:
an image sample library creating unit configured to create the image sample library, including pairs of the LR images and relevant HR images, by utilization of the LR images and the HR images.

### Claim 14 (depends on 13)

The resolving system according to claim 13, wherein the training device further includes:
a second training unit configured to obtain second filtering parameters of the convolutional structural network by analyzing a correlation between the pairs of the LR images and the relevant HR images; obtain a high-pass filtering result and a low-pass filtering result by respectively performing high-pass filtering and low-pass filtering on the HR images; obtain a detail template of the HR images as a feedback signal of the convolutional structural network by a superimposition and a feature classification of the high-pass filtering result and the low-pass filtering result; and adopt the LR images as an input signal of the convolutional structural network, adjust the second filtering parameters in the convolutional structural network, and obtain a forecast result signal, being the same with the feedback signal, by processing the input signal by utilization of the convolutional structural network according to the adjusted second filtering parameters.

### Claim 15 (depends on 14)

The resolving system according to claim 14, wherein the output device further includes:
an input unit further configured to input the face feature information and/or images with low resolution;
a convolutional structural network further configured to process the inputted face feature information of the LR images and/or the images by utilization of the convolutional structural network according to the adjusted first and/or second filtering parameters; and
an output unit further configured to output face feature information and/or images with high resolution, processed by the convolutional structural network.

### Claim 16 (depends on 15)

The resolving system according to claim 15, wherein each filter unit adopts a formula F(x)=Wx+b to execute a convolution operation, where W and b refer to the filtering parameters; x refers to input; and F(x) refers to output.

### Claim 17 (depends on 16)

The resolving system according to claim 16, wherein the forecast result signal is determined to be the same with the feedback signal when J(W,b) obtained according to the following formula is less than a first threshold:
J
⁡
(
W
,
b
)
=
1
m
⁢
∑
i
=
1
m
⁢
(
1
2
⁢

h
W
,
b
⁡
(
I
HR
i
-
I
HR
_
i
)

2
)
where J(W,b) refers to a mean square error; m refers to a number of image sets in the face feature sample library; IHRi refers to the feedback signal; IHRi refers to the forecast result signal; and hW,b refers to a weight coefficient.

### Claim 18 (depends on 17)

The resolving system according to claim 17, wherein, when the forecast result signal is different from the feedback signal, a partial derivative of J(W,b) is calculated for each filtering parameter, and the first filtering parameters or the second filtering parameters are adjusted according to the partial derivatives.