# US 12,223,618 B2 · Targeted super-resolution image processing method and apparatus, and electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,618 B2
- Document ID: US12223618B2
- Status: Granted
- Kind: B2
- Application number: 17/726,218
- Filed: 2022-04-21
- Published: 2022-08-04
- Granted: 2025-02-11
- Earliest claimed: 2019-10-22
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Hengming Wang; Yang Wang; Jie Ji
- Web page: https://patentgrep.com/patent/US12223618B2

## CPC

- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G09G 5/005: {Adapting incoming signals to the display format of the display terminal}
- G09G 5/391: Resolution modifying circuits, e.g. variable screen formats
- H04N 19/132: Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
- H04N 19/176: the region being a block, e.g. a macroblock
- H04N 23/951: by using two or more images to influence resolution, frame rate or aspect ratio
- G06T 2207/10016: Video; Image sequence
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G09G 2340/0407: Resolution change, inclusive of the use of different resolutions for different screen areas
- G09G 2370/027: Arrangements and methods specific for the display of internet documents
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

The present application discloses an image processing method and apparatus, and an electronic device, and pertains to the field of image processing. The method includes: obtaining an inter-frame residual between two adjacent image frames to obtain a residual block; determining, based on the residual block, a target pixel region on which super-resolution processing needs to be performed; and performing super-resolution processing on the target pixel region to obtain a target pixel region. For another pixel region, the super-resolution processing result of an image frame that is in the two image frames and on which super-resolution processing is performed is used. This application reduces high computation costs of super-resolution processing.

## Claims

### Claim 1

An image processing method, wherein the method comprises:
obtaining an inter-frame residual between a first image and a previous image frame adjacent to the first image to obtain a residual block, wherein the residual block comprises a plurality of residual points that are in a one-to-one correspondence with locations of a plurality of pixels in the first image, and each residual point has one residual value;
determining a first target pixel region in the first image based on the residual block;
performing super-resolution processing on the first target pixel region in the first image to obtain a second target pixel region after the super-resolution processing; and
updating a first pixel region in the first image by using a second pixel region that is in the previous image frame and that is obtained after super-resolution processing, wherein
the first pixel region comprises a pixel region in the first image other than the first target pixel region, and
the first image obtained after super-resolution processing comprises the second target pixel region obtained after the super-resolution processing and the updated first pixel region.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the determining of the first target pixel region in the first image based on the residual block comprises:
generating a mask pattern based on the residual block, wherein the mask pattern comprises a plurality of first mask points, and the plurality of first mask points are in a one-to-one correspondence with locations of a plurality of target residual points in the residual block; and
inputting the mask pattern and the first image into a super-resolution model, and determining, as the first target pixel region by using the super-resolution model, a region that is in the first image and in which pixels corresponding to locations of all of the plurality of first mask points are located.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the performing of the super-resolution processing on the first target pixel region in the first image to obtain the second target pixel region after the super-resolution processing comprises:
performing the super-resolution processing on the first target pixel region in the first image by using the super-resolution model, to obtain the second target pixel region after the super-resolution processing.

### Claim 4 (depends on 2)

The method according to claim 2, wherein the generating of the mask pattern based on the residual block comprises:
generating, based on the residual block, an initial mask pattern comprising a plurality of mask points, wherein the plurality of mask points are in a one-to-one correspondence with locations of the plurality of pixels in the first image, and the plurality of mask points comprise the plurality of first mask points and a plurality of second mask points; and
assigning a first value to a mask value of a first mask point in the initial mask pattern, and assigning a second value to a mask value of a second mask point in the mask pattern, to obtain the mask pattern, wherein the first value is different from the second value; and
the determining, as the first target pixel region, the pixels that are in the first image and that correspond to the locations of the plurality of first mask points comprises:
traversing mask points in the mask pattern, and determining, in the first image, a pixel corresponding to a mask point whose mask value is the first value as the first target pixel region.

### Claim 5 (depends on 2)

The method according to claim 2, wherein the generating of the mask pattern based on the residual block comprises:
performing morphological transformation processing on the residual block to obtain the mask pattern, wherein the morphological transformation processing comprises binarization processing and dilation processing that are performed on a first mask point in a residual block obtained after the binarization, the super-resolution model comprises at least one convolutional layer, and a kernel for the dilation processing has a same size as a receptive field of the last convolutional layer of the super-resolution model.

### Claim 6 (depends on 2)

The method according to claim 2, wherein the generating of the mask pattern based on the residual block comprises:
dividing the residual block into a plurality of residual sub-blocks, and performing block division processing on each residual sub-block obtained through the division, wherein the block division processing comprises:
when a residual value of at least one residual point comprised in the residual sub-block is not 0, dividing the residual sub-block into a plurality of residual sub-blocks, and performing block division processing on each residual sub-block obtained through the division, until residual values of all residual points comprised in a residual sub-block obtained through the division are 0, a total quantity of residual points in a residual sub-block obtained through the division is less than a point quantity threshold, or a total quantity of times for dividing the residual block reaches a quantity of times threshold; and
generating a mask sub-pattern corresponding to each target residual block, wherein a residual value of at least one residual point comprised in the target residual block is not 0, and
the mask pattern comprises the generated mask sub-pattern.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the performing super-resolution processing on the first target pixel region in the first image to obtain the second target pixel region after the super-resolution processing comprises:
obtaining, from the first image, a target image block corresponding to each of the mask sub-patterns; and
performing super-resolution processing on a sub-region that is of the first target pixel region and that is comprised in each of the target image blocks, to obtain the second target pixel region after the super-resolution processing, wherein the second target pixel region obtained after the super-resolution processing comprises sub-regions of the first target pixel region that are comprised in all the target image blocks and that are obtained after the super-resolution processing.

### Claim 8 (depends on 1)

The method according to claim 1, wherein the determining of the target pixel region in the first image based on the residual block comprises:
generating a mask pattern based on the residual block, wherein the mask pattern comprises a plurality of first mask points, and the plurality of first mask points are in a one-to-one correspondence with locations of a plurality of target residual points in the residual block; and
determining, as the first target pixel region, a region that is in the first image and in which pixels corresponding to locations of all of the plurality of first mask points are located.

### Claim 9 (depends on 8)

The method according to claim 8, wherein the performing of the super-resolution processing on the first target pixel region in the first image to obtain the second target pixel region after the super-resolution processing comprises:
inputting the first target pixel region in the first image into a super-resolution model, and performing the super-resolution processing on the first target pixel region in the first image by using the super-resolution model, to obtain the second target pixel region after the super-resolution processing.

### Claim 10

An image processing apparatus, comprising a memory and a processor, wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program from the memory and run the computer program to perform an image processing method, the method comprising:
obtaining an inter-frame residual between a first image and a previous image frame adjacent to the first image to obtain a residual block, wherein the residual block comprises a plurality of residual points that are in a one-to-one correspondence with locations of a plurality of pixels in the first image, and each residual point has one residual value;
determining a first target pixel region in the first image based on the residual block;
performing super-resolution processing on the first target pixel region in the first image to obtain a second target pixel region after the super-resolution processing; and
updating a first pixel region in the first image by using a second pixel region that is in the previous image frame and that is obtained after super-resolution processing, wherein the first pixel region comprises a pixel region in the first image other than the first target pixel region, and
the first image obtained after super-resolution processing comprises the second target pixel region obtained after the super-resolution processing and the updated first pixel region.

### Claim 11 (depends on 10)

The apparatus according to claim 10, wherein the determining of the first target pixel region in the first image based on the residual block comprises:
generating a mask pattern based on the residual block, wherein the mask pattern comprises a plurality of first mask points, and the plurality of first mask points are in a one-to-one correspondence with locations of a plurality of target residual points in the residual block; and
inputting the mask pattern and the first image into a super-resolution model, and determining, as the first target pixel region by using the super-resolution model, a region that is in the first image and in which pixels corresponding to locations of all of the plurality of first mask points are located.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein the performing of the super-resolution processing on the first target pixel region in the first image to obtain the second target pixel region after the super-resolution processing comprises:
performing the super-resolution processing on the first target pixel region in the first image by using the super-resolution model, to obtain the second target pixel region after the super-resolution processing.

### Claim 13 (depends on 11)

The apparatus according to claim 11, wherein the generating of the mask pattern based on the residual block comprises:
generating, based on the residual block, an initial mask pattern comprising a plurality of mask points, wherein the plurality of mask points are in a one-to-one correspondence with locations of the plurality of pixels in the first image, and the plurality of mask points comprise the plurality of first mask points and a plurality of second mask points; and
assigning a first value to a mask value of a first mask point in the initial mask pattern, and assigning a second value to a mask value of a second mask point in the mask pattern, to obtain the mask pattern, wherein the first value is different from the second value; and
the determining, as the first target pixel region, the pixels that are in the first image and that correspond to the locations of the plurality of first mask points comprises:
traversing mask points in the mask pattern, and determining, in the first image, a pixel corresponding to a mask point whose mask value is the first value as the first target pixel region.

### Claim 14 (depends on 11)

The apparatus according to claim 11, wherein the generating of the mask pattern based on the residual block comprises:
performing morphological transformation processing on the residual block to obtain the mask pattern, wherein the morphological transformation processing comprises binarization processing and dilation processing that are performed on a first mask point in a residual block obtained after the binarization, the super-resolution model comprises at least one convolutional layer, and a kernel for the dilation processing has a same size as a receptive field of the last convolutional layer of the super-resolution model.

### Claim 15 (depends on 11)

The apparatus according to claim 11,
wherein the generating of the mask pattern based on the residual block comprises:
dividing the residual block into a plurality of residual sub-blocks, and performing block division processing on each residual sub-block obtained through the division, wherein the block division processing comprises:
when a residual value of at least one residual point comprised in the residual sub-block is not 0, dividing the residual sub-block into a plurality of residual sub-blocks, and performing block division processing on each residual sub-block obtained through the division, until residual values of all residual points comprised in a residual sub-block obtained through the division are 0, a total quantity of residual points in a residual sub-block obtained through the division is less than a point quantity threshold, or a total quantity of times for dividing the residual block reaches a quantity of times threshold; and
generating a mask sub-pattern corresponding to each target residual block, wherein a residual value of at least one residual point comprised in the target residual block is not 0, and
the mask pattern comprises the generated mask sub-pattern.

### Claim 16 (depends on 15)

The apparatus according to claim 15, wherein the performing of the super-resolution processing on the first target pixel region in the first image to obtain the second target pixel region after the super-resolution processing comprises:
obtaining, from the first image, a target image block corresponding to each of the mask sub-patterns; and
performing super-resolution processing on a sub-region that is of the first target pixel region and that is comprised in each of the target image blocks, to obtain the second target pixel region after the super-resolution processing, wherein the second target pixel region obtained after the super-resolution processing comprises sub-regions of the first target pixel region that are comprised in all the target image blocks and that are obtained after the super-resolution processing.

### Claim 17 (depends on 10)

The apparatus according to claim 10, wherein the determining of the target pixel region in the first image based on the residual block comprises:
generating a mask pattern based on the residual block, wherein the mask pattern comprises a plurality of first mask points, and the plurality of first mask points are in a one-to-one correspondence with locations of a plurality of target residual points in the residual block; and
determining, as the first target pixel region, a region that is in the first image and in which pixels corresponding to locations of all of the plurality of first mask points are located.

### Claim 18 (depends on 10)

The apparatus according to claim 10, wherein the performing of the super-resolution processing on the first target pixel region in the first image to obtain the second target pixel region after the super-resolution processing comprises:
inputting the first target pixel region in the first image into a super-resolution model, and performing the super-resolution processing on the first target pixel region in the first image by using the super-resolution model, to obtain the second target pixel region after the super-resolution processing.

### Claim 19

An electronic device, comprising a processor and a memory, wherein
the memory is configured to store a computer program; and
when the processor executes the computer program stored in the memory, causes the processor to perform an image processing method comprising:
obtaining an inter-frame residual between a first image and a previous image frame adjacent to the first image to obtain a residual block, wherein the residual block comprises a plurality of residual points that are in a one-to-one correspondence with locations of a plurality of pixels in the first image, and each residual point has one residual value;
determining a first target pixel region in the first image based on the residual block;
performing super-resolution processing on the first target pixel region in the first image to obtain a second target pixel region after the super-resolution processing; and
updating a first pixel region in the first image by using a second pixel region that is in the previous image frame and that is obtained after super-resolution processing, wherein the first pixel region comprises a pixel region in the first image other than the first target pixel region, and
the first image obtained after super-resolution processing comprises the second target pixel region obtained after the super-resolution processing and the updated first pixel region.

### Claim 20 (depends on 19)

The electronic device according to claim 19, wherein the determining of the first target pixel region in the first image based on the residual block comprises:
generating a mask pattern based on the residual block, wherein the mask pattern comprises a plurality of first mask points, and the plurality of first mask points are in a one-to-one correspondence with locations of a plurality of target residual points in the residual block; and
inputting the mask pattern and the first image into a super-resolution model, and determining, as the first target pixel region by using the super-resolution model, a region that is in the first image and in which pixels corresponding to locations of all of the plurality of first mask points are located.