# US 2021/0241470 A1 · Image processing method and apparatus, electronic device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241470 A1
- Document ID: US20210241470A1
- Status: Published application
- Kind: A1
- Application number: 17/236,023
- Filed: 2021-04-21
- Published: 2021-08-05
- Earliest claimed: 2019-04-30
- Applicants: Beijing Sensetime Technology Development Co., Ltd.
- Inventors: Xiaoou TANG; Xintao WANG; Zhuojie CHEN; Ke YU; Chao DONG; Chen Change LOY
- Web page: https://patentgrep.com/patent/US20210241470A1

## CPC

- G06F 18/253: {of extracted features}
- G06F 18/251: {of input or preprocessed data}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/73: Deblurring; Sharpening
- G06T 7/33: using feature-based methods
- G06V 10/443: {by matching or filtering}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/806: {of extracted features}
- G06V 10/82: using neural networks
- G06V 20/46: {Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames}
- G06V 20/48: {Matching video sequences}
- G06N 3/045: Combinations of networks
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20221: Image fusion; Image merging
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image processing method includes: acquiring an image frame sequence, including a to-be-processed image frame and one or more image frames adjacent thereto, and performing image alignment on the to-be-processed image frame and each of image frames in the image frame sequence to obtain multiple pieces of aligned feature data; determining, based on the multiple pieces of alignment feature data, multiple similarity features each between a respective one of the multiple pieces of aligned feature data and aligned feature data corresponding to the to-be-processed image frame, and determining weight information of each of multiple pieces of aligned feature data based on the multiple similarity features; and fusing the multiple pieces of aligned feature data according to the weight information to obtain fusion information of the image frame sequence, the fusion information being configured to acquire a processed image frame corresponding to the to-be-processed image frame.

## Claims

### Claim 1

A method for image processing, comprising:
acquiring an image frame sequence, comprising an image frame to be processed and one or more image frames adjacent to the image frame to be processed, and performing image alignment on the image frame to be processed and each of image frames in the image frame sequence to obtain a plurality of pieces of aligned feature data;
determining, based on the plurality of pieces of aligned feature data, a plurality of similarity features, each between a respective one of the plurality of pieces of aligned feature data and aligned feature data corresponding to the image frame to be processed, and determining, based on the plurality of similarity features, weight information of each of the plurality of pieces of aligned feature data; and
fusing the plurality of pieces of aligned feature data according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence, the fused information being configured to acquire a processed image frame corresponding to the image frame to be processed.

### Claim 2 (depends on 1)

The method for image processing of claim 1, wherein performing image alignment on the image frame to be processed and each of the image frames in the image frame sequence to obtain the plurality of pieces of aligned feature data comprises:
performing, based on a first image feature set and one or more second image feature sets, image alignment on the image frame to be processed and each of the image frames in the image frame sequence to obtain the plurality of pieces of aligned feature data, wherein:
the first image feature set comprises at least one piece of feature data of the image frame to be processed, and each of the at least one piece of feature data in the first image feature set has a respective different scale; and
each of the one or more second image feature sets comprises at least one piece of feature data of a respective image frame in the image frame sequence, and each of the at least one piece of feature data in the second image feature set has a respective different scale.

### Claim 3 (depends on 2)

The method for image processing of claim 2, wherein performing, based on the first image feature set and the one or more second image feature sets, image alignment on the image frame to be processed and each of the image frames in the image frame sequence to obtain the plurality of pieces of aligned feature data comprises:
action a), acquiring first feature data of a smallest scale in the first image feature set, and acquiring second feature data, of the same scale as the first feature data, in one of the one or more second image feature sets;
action b), performing image alignment on the first feature data and the second feature data to obtain first aligned feature data;
action c), acquiring third feature data of a second smallest scale in the first image feature set, and acquiring fourth feature data, of the same scale as the third feature data, in the second image feature set;
action d), performing upsampling convolution on the first aligned feature data to obtain the first aligned feature data having the same scale as that of the third feature data;
action e), performing, based on the first aligned feature data having subjected to the upsampling convolution, image alignment on the third feature data and the fourth feature data to obtain second aligned feature data;
action f), executing the actions a) to e) in a small-to-large order of scales until a piece of aligned feature data of the same scale as the image frame to be processed is obtained; and
action g), executing the actions a)-f) based on all the second image feature sets to obtain the plurality of pieces of aligned feature data.

### Claim 4 (depends on 3)

The method for image processing of claim 3, wherein after obtaining the plurality of pieces of aligned feature data, the method further comprises:
adjusting each of the plurality of pieces of aligned feature data based on a deformable convolutional network (DCN) to obtain a plurality pieces of adjusted aligned feature data.

### Claim 5 (depends on 1)

The method for image processing of claim 1, wherein determining, based on the plurality of pieces of aligned feature data, the plurality of similarity features, each between a respective one of the plurality of pieces of aligned feature data and the aligned feature data corresponding to the image frame to be processed comprises:
executing a dot product operation on each of the plurality of pieces of aligned feature data and the aligned feature data corresponding to the image frame to be processed, to determine the plurality of similarity features, each between a respective one of the plurality of pieces of aligned feature data and the aligned feature data corresponding to the image frame to be processed.

### Claim 6 (depends on 5)

The method for image processing of claim 5, wherein determining, based on the plurality of similarity features, the weight information of each of the plurality of pieces of aligned feature data comprises:
determining the weight information of each of the plurality of pieces of aligned feature data by a preset activation function and the plurality of similarity features, each between a respective one of the plurality of pieces of aligned feature data and the aligned feature data corresponding to the image frame to be processed.

### Claim 7 (depends on 1)

The method for image processing of claim 1, wherein fusing the plurality of pieces of aligned feature data according to the weight information of each of the plurality of pieces of aligned feature data, to obtain the fused information of the image frame sequence comprises:
fusing, by a fusion convolutional network, the plurality of pieces of aligned feature data according to the weight information of each of the plurality of pieces of aligned feature data, to obtain the fused information of the image frame sequence.

### Claim 8 (depends on 7)

The method for image processing of claim 7, wherein fusing, by the fusion convolutional network, the plurality of pieces of aligned feature data according to the weight information of each of the plurality of pieces of aligned feature data, to obtain the fused information of the image frame sequence comprises:
multiplying, through element-wise multiplication, each of the plurality of pieces of aligned feature data by a respective piece of weight information, to obtain a plurality pieces of modulated feature data, each for a respective one of the plurality of pieces of aligned feature data; and
fusing, by the fusion convolutional network, the plurality pieces of modulated feature data to obtain the fused information of the image frame sequence.

### Claim 9 (depends on 7)

The method for image processing of claim 7, wherein after fusing, by the fusion convolutional network, the plurality of pieces of aligned feature data according to the weight information of each of the plurality of pieces of aligned feature data, to obtain the fused information of the image frame sequence, the method further comprises:
generating spatial feature data based on the fused information of the image frame sequence; and
modulating the spatial feature data based on spatial attention information of each element in the spatial feature data to obtain modulated fused information, the modulated fused information being configured to acquire the processed image frame corresponding to the image frame to be processed.

### Claim 10 (depends on 9)

The method for image processing of claim 9, wherein modulating the spatial feature data based on the spatial attention information of each element in the spatial feature data to obtain the modulated fused information comprises:
modulating, by element-wise multiplication and addition, each element in the spatial feature data according to respective spatial attention information of the element in the spatial feature data, to obtain the modulated fused information.

### Claim 11 (depends on 1)

The method for image processing of claim 1, wherein the method for image processing is implemented based on a neural network; and
the neural network is obtained by training with a dataset comprising a plurality of sample image frame pairs, each of the sample image frame pairs comprises a first sample image frame and a second sample image frame corresponding to the first sample image frame, and a resolution of the first sample image frame is lower than a resolution of the second sample image frame.

### Claim 12 (depends on 1)

The method for image processing of claim 1, wherein before acquiring the image frame sequence, the method further comprises:
subsampling each video frame in an acquired video sequence to obtain the image frame sequence.

### Claim 13 (depends on 1)

The method for image processing of claim 1, wherein before performing image alignment on the image frame to be processed and each of the image frames in the image frame sequence, the method further comprises:
performing deblurring on the image frames in the image frame sequence.

### Claim 14 (depends on 1)

The method for image processing of claim 1, further comprising:
acquiring, according to the fused information of the image frame sequence, the processed image frame corresponding to the image frame to be processed.

### Claim 15 (depends on 1)

A method for image processing, comprising:
in response to that a resolution of an image frame sequence in a first video stream acquired by a video acquisition device is less than or equal to a preset threshold value, sequentially processing each image frame in the image frame sequence through the method of claim 1 to obtain a processed image frame sequence; and
performing at least one of: outputting or displaying a second video stream formed by the processed image frame sequence.

### Claim 16

An electronic device, comprising a processor and a memory, wherein the memory is configured to store instructions which, when being executed by the processor, cause the processor to carry out the following:
acquiring an image frame sequence, comprising an image frame to be processed and one or more image frames adjacent to the image frame to be processed, and performing image alignment on the image frame to be processed and each of image frames in the image frame sequence to obtain a plurality of pieces of aligned feature data;
determining, based on the plurality of pieces of aligned feature data, a plurality of similarity features, each between a respective one of the plurality of pieces of aligned feature data and aligned feature data corresponding to the image frame to be processed, and determining, based on the plurality of similarity features, weight information of each of the plurality of pieces of aligned feature data; and
fusing the plurality of pieces of aligned feature data according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence, the fused information being configured to acquire a processed image frame corresponding to the image frame to be processed.

### Claim 17 (depends on 16)

The electronic device of claim 16, wherein in performing image alignment on the image frame to be processed and each of the image frames in the image frame sequence to obtain the plurality of pieces of aligned feature data, the processor is caused to carry out the following:
performing, based on a first image feature set and one or more second image feature sets, image alignment on the image frame to be processed and each of the image frames in the image frame sequence to obtain the plurality of pieces of aligned feature data, wherein:
the first image feature set comprises at least one piece of feature data of the image frame to be processed, and each of the at least one piece of feature data in the first image feature set has a respective different scale; and
each of the one or more second image feature sets comprises at least one piece of feature data of a respective image frame in the image frame sequence, and each of the at least one piece of feature data in the second image feature set has a respective different scale.

### Claim 18 (depends on 17)

The electronic device of claim 17, wherein in performing, based on the first image feature set and the one or more second image feature sets, image alignment on the image frame to be processed and each of the image frames in the image frame sequence to obtain the plurality of pieces of aligned feature data, the processor is caused to perform the following:
action a), acquiring first feature data of a smallest scale in the first image feature set, and acquiring second feature data, of the same scale as the first feature data, in one of the one or more second image feature sets;
action b), performing image alignment on the first feature data and the second feature data to obtain first aligned feature data;
action c), acquiring third feature data of a second smallest scale in the first image feature set, and acquiring fourth feature data, of the same scale as the third feature data, in the second image feature set;
action d), performing upsampling convolution on the first aligned feature data to obtain the first aligned feature data having the same scale as that of the third feature data;
action e), performing, based on the first aligned feature data having subjected to the upsampling convolution, image alignment on the third feature data and the fourth feature data to obtain second aligned feature data;
action f), executing the actions a) to e) in a small-to-large order of scales until a piece of aligned feature data of the same scale as the image frame to be processed is obtained; and
action g), executing the actions a)-f) based on all the second image feature sets to obtain the plurality of pieces of aligned feature data.

### Claim 19 (depends on 18)

The electronic device of claim 18, wherein the processor is caused to carry out the following:
after obtaining the plurality of pieces of aligned feature data, adjusting each of the plurality of pieces of aligned feature data based on a deformable convolutional network (DCN) to obtain a plurality pieces of adjusted aligned feature data.

### Claim 20

A non-transitory computer-readable storage medium, configured to store instructions which, when being executed by a processor, cause the processor to carry out the following:
acquiring an image frame sequence, comprising an image frame to be processed and one or more image frames adjacent to the image frame to be processed, and performing image alignment on the image frame to be processed and each of image frames in the image frame sequence to obtain a plurality of pieces of aligned feature data;
determining, based on the plurality of pieces of aligned feature data, a plurality of similarity features, each between a respective one of the plurality of pieces of aligned feature data and aligned feature data corresponding to the image frame to be processed, and determining, based on the plurality of similarity features, weight information of each of the plurality of pieces of aligned feature data; and
fusing the plurality of pieces of aligned feature data according to the weight information of each of the plurality of pieces of aligned feature data, to obtain fused information of the image frame sequence, the fused information being configured to acquire a processed image frame corresponding to the image frame to be processed.