# US 10,121,227 B1 · Method and system of reconstructing videos by using super-resolution algorithm

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,121,227 B1
- Document ID: US10121227B1
- Status: Granted
- Kind: B1
- Application number: 15/709,558
- Filed: 2017-09-20
- Granted: 2018-11-06
- Earliest claimed: 2017-08-31
- Assignees: ULSee Inc.
- Applicants: ULSee Inc.
- Inventors: Yang Liu
- Web page: https://patentgrep.com/patent/US10121227B1

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 3/4007: based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403)
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- H04N 7/0117: {involving conversion of the spatial resolution of the incoming video signal (for graphics images G09G2340/0407)}
- H04N 7/0125: {one of the standards being a high definition standard}
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20221: Image fusion; Image merging
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method of reconstructing videos by using super-resolution algorithm includes the steps: (a) providing a video, wherein the video is composed of a plurality of frames having a sequence; (b) starting a first thread and a second thread, the first thread performing a first algorithm and the second thread performing a second algorithm for improving resolution, wherein a time complexity of the first algorithm is greater than a time complexity of the second algorithm; (c) the first thread sequentially reading the frames of the video in units of a first interval and processing the frames in order to obtain first processed frames, and the second thread sequentially reading the frames of the video in units of a second interval and processing the frames in order to obtain second processed frames, wherein a value of the first interval is an integer greater than 1, and a value of the second interval is 1; (d) performing a fusion operation on the second processed frames which is processed by the second thread and the nearest first processed frames which is processed by the first thread in order to obtain a plurality of fusion frames; and (e) displaying the plurality of fusion frames according to the sequence. The present disclosure can solve the problem that video super-resolution in real-time, computing time and resolution are difficult to be taken into account simultaneously.

## Claims

### Claim 1

A method of reconstructing videos by using super-resolution algorithm, comprising:
(a) providing a video, wherein the video is composed of a plurality of frames having a sequence;
(b) starting a first thread and a second thread, the first thread performing a first algorithm and the second thread performing a second algorithm for improving resolution, wherein a time complexity of the first algorithm is greater than a time complexity of the second algorithm;
(c) the first thread sequentially reading the frames of the video in units of a first interval and processing the frames in order to obtain first processed frames, and the second thread sequentially reading the frames of the video in units of a second interval and processing the frames in order to obtain second processed frames, wherein a value of the first interval is an integer greater than 1, and a value of the second interval is 1;
(d) performing a fusion operation on the second processed frames which is processed by the second thread and the nearest first processed frames which is processed by the first thread in order to obtain a plurality of fusion frames; and
(e) displaying the plurality of fusion frames according to the sequence;
wherein the following formula is used for obtaining the fusion frames in the step (d):
F2F=A·F1H+B·F2L;
wherein F2L represents the second processed frames which is processed by the second thread, F1H represent the first processed frames which is processed by the first thread and nearest to the second processed frames F2L, A, B∈Rm×n, and A and B meet the following relationship:
{
A
+
B
=
I
B
=
f
⁡
(

F
2
⁢
L
-
F
1
⁢
L

)
;
wherein F1L and F1H correspond to a same original frame, F1L is obtained from processing the original frame by the second thread, I is an m×n matrix with all elements of 1, and the function f(x) is expressed as follows:
f
⁡
(
x
)
=
1
1
+
e
-
a
⁡
(
x
+
b
)
.

### Claim 2 (depends on 1)

The method of reconstructing videos by using super-resolution algorithm according to claim 1, further comprising the following step between the step (d) and the step (e):
storing the plurality of fusion frames until the video ends.

### Claim 3 (depends on 1)

The method of reconstructing videos by using super-resolution algorithm according to claim 1, wherein the video in the step (a) is a real-time video; and in the step (e), each of the plurality of fusion frames obtained in the step (d) is displayed in real-time.

### Claim 4 (depends on 1)

The method of reconstructing videos by using super-resolution algorithm according to claim 1, wherein values of a and b are expressed by the following relationship:
{
a
=
1
var
⁡
(

F
2
⁢
L
-
F
1
⁢
L

)
b
=
0.03
+
log
⁡
(
1
0.97
-
1
)
a
.

### Claim 5 (depends on 4)

The method of reconstructing videos by using super-resolution algorithm according to claim 4, wherein f(0.03)=0.97.

### Claim 6 (depends on 1)

The method of reconstructing videos by using super-resolution algorithm according to claim 1, wherein the first algorithm is a deep-learning-based super-resolution algorithm, and the second algorithm is a bi-cubic interpolation.

### Claim 7

A system of reconstructing videos by using super-resolution algorithm, the system comprising:
a video providing module, used for providing a video, wherein the video is composed of a plurality of frames having a sequence;
a thread starting module, used for starting a first thread and a second thread, the first thread performing a first algorithm and the second thread performing a second algorithm for improving resolution, wherein a time complexity of the first algorithm is greater than a time complexity of the second algorithm;
the first thread, used for sequentially reading the frames of the video in units of a first interval and processing the frames in order to obtain first processed frames;
the second thread, used for sequentially reading the frames of the video in units of a second interval and processing the frames in order to obtain second processed frames, wherein a value of the first interval is an integer greater than 1, and a value of the second interval is 1;
a fusion operation module, used for performing a fusion operation on the second processed frames which is processed by the second thread and the nearest first processed frames which is processed by the first thread in order to obtain a plurality of fusion frames; and
a displaying module, used for displaying the plurality of fusion frames according to the sequence;
wherein the fusion operation module adopts the following formula for obtaining the fusion frames:
F2F=A·F1H+B·F2L;
wherein F2L represents the second processed frames which is processed by the second thread, F1H represent the first processed frames which is processed by the first thread and nearest to the second processed frames F2L, A, B∈Rm×n, and A and B meet the following relationship:
{
A
+
B
=
I
B
=
f
⁡
(

F
2
⁢
L
-
F
1
⁢
L

)
;
wherein F1L and F1H correspond to a same original frame, F1L is obtained from processing the original frame by the second thread, I is an m×n matrix with all elements of 1, and the function f(x) is expressed as follows:
f
⁡
(
x
)
=
1
1
+
e
-
a
⁡
(
x
+
b
)
.

### Claim 8 (depends on 7)

The system of reconstructing videos by using super-resolution algorithm according to claim 7, further comprising:
a storage, used for storing the plurality of fusion frames until the video ends.

### Claim 9 (depends on 7)

The system of reconstructing videos by using super-resolution algorithm according to claim 7, wherein the video provided by the video providing module is a real-time video; and each of the plurality of fusion frames obtained by the fusion operation module is displayed in real-time.

### Claim 10 (depends on 7)

The system of reconstructing videos by using super-resolution algorithm according to claim 7, wherein values of a and b are expressed by the following relationship:
{
a
=
1
var
⁡
(

F
2
⁢
L
-
F
1
⁢
L

)
b
=
0.03
+
log
⁡
(
1
0.97
-
1
)
a
.

### Claim 11 (depends on 10)

The system of reconstructing videos by using super-resolution algorithm according to claim 10, wherein f(0.03)=0.97.

### Claim 12 (depends on 7)

The system of reconstructing videos by using super-resolution algorithm according to claim 7, wherein the first algorithm is a deep-learning-based super-resolution algorithm, and the second algorithm is a bi-cubic interpolation.