# US 2023/0056332 A1 · Image Processing Method and Related Apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0056332 A1
- Document ID: US20230056332A1
- Status: Published application
- Kind: A1
- Application number: 17/784,342
- Filed: 2020-12-14
- Published: 2023-02-23
- Earliest claimed: 2019-12-13
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Gang Chen; Jiangwei Li; Wei Luo; Chunlin Qin; Zhen Wang; Jing Hu; Jinlei Zhang; Zheng Li
- Web page: https://patentgrep.com/patent/US20230056332A1

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 5/73: Deblurring; Sharpening
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06V 10/267: {by performing operations on regions, e.g. growing, shrinking or watersheds}
- G06V 10/759: {Region-based matching}
- G06V 10/761: {Proximity, similarity or dissimilarity measures}
- G06V 20/70: Labelling scene content, e.g. deriving syntactic or semantic representations
- G06T 2207/10004: Still image; Photographic image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image processing method includes segmenting, by a cloud server and based on a zoom ratio of a high-definition image in a high-definition image library, a low-definition image of a small zoom ratio from a terminal to obtain a plurality of image blocks. For each image block, performing, by the cloud server, retrieval and matching in the high-definition image library to obtain one or more high-definition reference images, and then performing, by the cloud server, image quality improvement using an image enhancement network that is based on the high-definition reference images to obtain a plurality of high-definition image blocks. Finally, stitching, by the cloud server, the high-definition image blocks into a super-definition image with higher resolution and higher definition, and returning, by the cloud server, the super-definition image to the terminal.

## Claims

### Claim 1

An image processing method implemented by a cloud server, wherein the image processing method comprises:
obtaining, from a terminal, a low-definition image and first photographing information that is for photographing the low-definition image, wherein the first photographing information comprises a first zoom ratio of the low-definition image;
identifying semantic information of the low-definition image;
determining, based on a second zoom ratio of a high-definition image in a high-definition image library and the first zoom ratio, a quantity of N image blocks, wherein N is a positive integer;
segmenting, based on the quantity of N image blocks, the low-definition image into N image blocks;
marking, based on the semantic information, a type of image content in each of the N image blocks;
retrieving, from the high-definition image library, first T retrieved high-definition images matching first photographed content of a first-type image block in the low-definition image, wherein T is a positive integer;
performing, based on the first T retrieved high-definition images, a first detail restoration on the first-type image block to obtain a first-type high-definition image block, wherein a first resolution of the first-type high-definition image block is greater than a second resolution of the first-type image block;
replacing, based on the first-type high-definition image block, the first-type image block to obtain a high-definition output image; and
sending, to the terminal, the high-definition output image.

### Claim 2 (depends on 1)

The image processing method of claim 1, wherein the first photographing information further comprises one or more of geographic location information, weather information, season information, time information, a photographing parameter, or photographing posture information.

### Claim 3 (depends on 1)

The image processing method of claim 1, further comprising:
determining, based on the first photographing information, at least one first matched image from the high-definition image library, wherein a degree of matching between second photographing information of the at least one first matched image and the first photographing information is greater than a first threshold, and wherein a third resolution of the at least one first matched image is greater than a fourth resolution of the low-definition image;
separately obtaining a first feature corresponding to each of the at least one first matched image and a second feature corresponding to the first-type image block, wherein one first matched image corresponds to one first feature or one second feature; and
determining, from the at least one first matched image based on a first similarity between the first feature and the second feature, second T retrieved high-definition images comprising second photographed content similar to the first photographed content.

### Claim 4 (depends on 3)

The image processing method of claim 3, further comprising:
determining, based on the first similarity, at least one second matched image comprising a third feature similar to the second feature;
separately obtaining a fourth feature corresponding to each of the at least one second matched image and a fifth feature corresponding to the first-type image block, wherein one second matched image corresponds to one fourth feature or one fifth feature; and
determining, based on a second similarity between the fourth feature and the fifth feature, third T retrieved high-definition images comprising sixth features similar to the fifth feature,
wherein the fourth feature, the fifth feature, and the sixth features are finer than the first feature, the second feature, and the third feature.

### Claim 5 (depends on 1)

The image processing method of claim 1, further comprising:
obtaining an image spatial coordinate transformation parameter between the first-type image block and each of the first T retrieved high-definition images;
adjusting, based on the image spatial coordinate transformation parameter, a first feature point that is in each of the first T retrieved high-definition images and that matches a second feature point in the first-type image block to a same location as that in the first-type image block to obtain second T registered high-definition images; and
performing, based on the second T registered high-definition images a second detail restoration on the first-type image block to obtain the first-type high-definition image block.

### Claim 6 (depends on 5)

The image processing method of claim 5, further comprising:
determining, from the first T registered high-definition images based on the image spatial coordinate transformation parameter, L high-definition reference images for which image spatial locations of matched feature point pairs are most similar, wherein L is a positive integer; and
performing, based on the L high-definition reference images, a third detail restoration on the first-type image block to obtain the first-type high-definition image block.

### Claim 7 (depends on 1)

The image processing method of claim 1, further comprising:
stitching second-type high-definition image blocks into the high-definition output image when the low-definition image comprises second-type image blocks; and
stitching a third-type high-definition image block and a fourth-type high-definition image block into the high-definition output image when the low-definition image comprises a third-type image block and a fourth-type image block.

### Claim 8 (depends on 1)

The image processing method of claim 1, wherein before segmenting the low-definition image into the N image blocks, the method further comprises:
determining, based on the first photographing information, an image processing model;
processing, using the processing model, the low-definition image to obtain a processed low-definition image; and
segmenting, based on the quantity of N image blocks, the processed low-definition image into the N image blocks.

### Claim 9 (depends on 8)

The image processing method of claim 8, further comprising processing, using the image processing model, a detail definition of the low-definition image, a dynamic range of the low-definition image, a contrast of the low-definition image, a noise of the low-definition image, or a color of the low-definition image.

### Claim 10

A cloud server comprising:
a memory configured to store computer instructions; and
a processor coupled to the memory, wherein when executed by the processor, the computer instructions cause the cloud server to:
obtain, from a terminal, a low-definition image and first photographing information that is used for photographing the low-definition image, wherein the first photographing information comprises a first zoom ratio of the low-definition image;
identify, semantic information of the low-definition image;
determine, based on a second zoom ratio of a high-definition image in a high-definition image library and the first zoom ratio, a quantity of N image blocks, wherein N is a positive integer;
segment, based on the quantity of N image blocks, the low-definition image into N image blocks;
mark, based on the semantic information, a type of image content in each of the N image blocks;
retrieve, from the high-definition image library, first T retrieved high-definition images matching first photographed content of a first-type image block in the low-definition image, wherein T is a positive integer;
perform, based on the first T retrieved high-definition images, a first detail restoration on the first-type image block to obtain a first-type high-definition image block, wherein a first resolution of the first-type high-definition image block is greater than a second resolution of the first-type image block; and
replace, based on the first-type high-definition image block, the first-type image block to obtain a high-definition output image; and
send, to the terminal, the high-definition output image.

### Claim 11

(canceled)

### Claim 12 (depends on 10)

The cloud server of claim 10, wherein the first photographing information further comprises one or more of geographic location information, weather information, season information, time information, a photographing parameter, or photographing posture information.

### Claim 13 (depends on 10)

The cloud server of claim 10, wherein when executed by the processor, the computer instructions further cause the cloud server to:
determine, based on the first photographing information, at least one first matched image from the high-definition image library, wherein a degree of matching between second photographing information of the at least one first matched image and the first photographing information is greater than a first threshold, and wherein a third resolution of the at least one first matched image is greater than a fourth resolution of the low-definition image;
separately obtain a first feature corresponding to each of the at least one first matched image and a second feature corresponding to the first-type image block, wherein one first matched image corresponds to one first feature or one second feature; and
determine, from the at least one first matched image based on a first similarity between the first feature and the second feature, second T retrieved high-definition images comprising second photographed content similar to the first photographed content.

### Claim 14 (depends on 13)

The cloud server of claim 13, wherein when executed by the processor, the computer instructions further cause the cloud server to:
determine, based on the first similarity, at least one second matched image comprising a third feature similar to the second feature;
separately obtain a fourth feature corresponding to each of the at least one second matched image and a fifth feature corresponding to the first-type image block, wherein one second matched image corresponds to one fourth feature or one fifth feature; and
determine, based on a second similarity between the fourth feature and the fifth feature, third T retrieved high-definition images comprising sixth features similar to the fifth feature,
wherein the fourth feature, the fifth feature, and the sixth features are finer than the first feature, the second feature, and the third feature.

### Claim 15 (depends on 10)

The cloud server of claim 10, wherein when executed by the processor, the computer instructions further cause the cloud server to:
obtain an image spatial coordinate transformation parameter between the first-type image block and each of the first T retrieved high-definition images;
adjust, based on the image spatial coordinate transformation parameter, a first feature point that is in each of the first retrieved high-definition images and that matches a second feature point in the first-type image block to a same location as that in the first-type image block to obtain second T registered high-definition images; and
perform, based on the second T registered high-definition images, a second detail restoration on the first-type image block to obtain the first-type high-definition image block.

### Claim 16 (depends on 15)

The cloud server of claim 15, wherein when executed by the processor, the computer instructions further cause the cloud server to:
determine, from the first T registered high-definition images based on the image spatial coordinate transformation parameter, L high-definition reference images for which image spatial locations of matched feature point pairs are most similar, wherein L is a positive integer; and
perform, based on the L high-definition reference images, a third detail restoration on the first-type image block to obtain the first-type high-definition image block.

### Claim 17 (depends on 10)

The cloud server of claim 10, wherein when executed by the processor, the computer instructions further cause the cloud server to:
stitch second-type high-definition image blocks into the high-definition output image when the low-definition image comprises second-type image blocks; and
stitch a third-type high-definition image block and a fourth-type high-definition image block into the high-definition output image when the low-definition image comprises a third-type image block and a fourth-type image block.

### Claim 18 (depends on 10)

The cloud server of claim 10, wherein when executed by the processor, the computer instructions further cause the cloud server to:
determine, based on the first photographing information, an image processing model;
process, using the image processing model, the low-definition image to obtain a processed low-definition image; and
segment, based on the quantity of N image blocks, the processed low-definition image into the N image blocks.

### Claim 19 (depends on 18)

The cloud server of claim 18, wherein when executed by the processor, the computer instructions further cause the cloud server to process, using the image processing model, a detail definition of the low-definition image, a dynamic range of the low-definition image, a contrast of the low-definition image, a noise of the low-definition image, or a color of the low-definition image.

### Claim 20

A computer program product comprising computer-executable instructions that are stored on a non-transitory computer-readable medium and that, when executed by a processor, cause a cloud server to:
obtain, from a terminal, a low-definition image and first photographing information that is used for photographing the low-definition image, wherein the first photographing information comprises a first zoom ratio of the low-definition image;
identify semantic information of the low-definition image;
determine, based on a second zoom ratio of a high-definition image in a high-definition image library and the first zoom ratio, a quantity of N image blocks, wherein N is a positive integer;
segment, based on the quantity of N image blocks, the low-definition image into N image blocks;
mark, based on the semantic information, a type of image content in each of the N image blocks;
retrieve, from the high-definition image library, first T retrieved high-definition images matching first photographed content of a first-type image block in the low-definition image, wherein T is a positive integer;
perform, based on the first T retrieved high-definition images, a first detail restoration on the first-type image block to obtain a first-type high-definition image block, wherein a first resolution of the first-type high-definition image block is greater than a second resolution of the first-type image block;
replace, based on the first-type high-definition image block, the first-type image block to obtain a high-definition output image; and
send, to the terminal, the high-definition output image.

### Claim 21 (depends on 20)

The computer program product of claim 20, wherein the first photographing information further comprises one or more of geographic location information, weather information, season information, time information, a photographing parameter, or photographing posture information.
