# US 12,223,616 B2 · Image processing method and apparatus, computer readable storage medium, and computer device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,223,616 B2
- Document ID: US12223616B2
- Status: Granted
- Kind: B2
- Application number: 17/727,042
- Filed: 2022-04-22
- Published: 2022-08-11
- Granted: 2025-02-11
- Earliest claimed: 2020-02-07
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Hongyun Gao
- Web page: https://patentgrep.com/patent/US12223616B2

## CPC

- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06T 3/4046: using neural networks
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method includes zooming out a to-be-processed image from a first size to a second size and determining a first feature map based on at least a feature reconstruction on the to-be-processed image of the second size by a first codec of a first set of coding units. The first set of coding units includes at least a pair of encoder and decoder. The method includes zooming in the first feature map to the first size, performing a splicing processing on the to-be-processed image of the first size and the first feature map of the first size to obtain a spliced feature map, and performing a feature reconstruction on the spliced feature map by a target codec, to obtain a target image. The target codec includes a target set of coding units, and a first subset of the target set corresponds to the first set of coding units.

## Claims

### Claim 1

A method of image processing, comprising:
acquiring a to-be-processed image;
zooming out the to-be-processed image from a first size to a second size;
performing a splicing processing on a first output feature map from a first processing unit and the to-be-processed image of the second size to obtain a first spliced output feature map for inputting into a second processing unit of a first codec;
determining, by processing circuitry of an electronic device, a first feature map based on at least a feature reconstruction on the to-be-processed image of the second size by the first codec of a first set of coding units, the first set of coding units comprising at least a pair of encoder and decoder, a coding unit in the first codec including at least the first processing unit and the second processing unit;
zooming in the first feature map to the first size;
performing a splicing processing on the to-be-processed image of the first size and the first feature map of the first size to obtain a spliced feature map; and
performing a feature reconstruction on the spliced feature map by a target codec, to obtain a target image, the target codec comprising a target set of coding units, a first subset of the target set of coding units corresponding to the first set of coding units.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the determining the first feature map comprises:
performing the feature reconstruction on the to-be-processed image of the second size by the first codec to obtain a coarse feature map of the second size;
zooming in the coarse feature map of the second size to obtain a zoomed coarse feature map of a third size between the first size and the second size;
zooming out the to-be-processed image of the first size to the third size;
performing a splicing processing on the zoomed coarse feature map of the third size and the to-be-processed image of the third size to obtain a second feature map; and
performing a feature reconstruction on the second feature map by a second codec, to obtain the first feature map, the second codec comprising a second set of coding units, a second subset of the target set of coding units corresponding to the second set of coding units, and a subset of the second set of coding units corresponding to the first set of coding units.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the performing splicing processing on the zoomed coarse feature map of the third size and the to-be-processed image of the third size to obtain the second feature map comprises:
combining the zoomed coarse feature map of the third size and the to-be-processed image of the third size in parallel in a channel dimension to obtain the second feature map.

### Claim 4 (depends on 1)

The method according to claim 1, further comprising:
generating the first output feature map by the first processing unit based on a first input feature map; and
performing a fusion processing on the first output feature map of the first processing unit and the first input feature map of the first processing unit to obtain a first fused feature map for inputting to the second processing unit of the first codec.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the coding unit of the first codec comprises a processing unit that generates an output feature map in response to an input feature map by a plurality of convolution layers, and the method comprises:
generating, by a first convolution layer in the plurality of convolution layers, a first intermediate feature map in response to the input feature map and a second intermediate feature map output from a second convolution layer in the plurality of convolution layers; and
performing a splicing processing on the input feature map of the processing unit and intermediate feature map outputs from the plurality of convolution layers to obtain the output feature map of the processing unit.

### Claim 6 (depends on 2)

The method according to claim 2, further comprising:
performing a splicing processing on the first output feature map from the first processing unit and the to-be-processed image of the second size to obtain the first spliced output feature map for inputting into the second processing unit of the first codec.

### Claim 7 (depends on 2)

The method according to claim 2, further comprising:
generating the first output feature map by the first processing unit based on a first input feature map; and
performing a fusion processing on the first output feature map of the first processing unit and the first input feature map of the first processing unit to obtain a first fused feature map for inputting to the second processing unit of the first codec.

### Claim 8 (depends on 2)

The method according to claim 2, wherein the coding unit in at least one of the first codec or the second codec comprises a processing unit that generates an output feature map in response to an input feature map based on a plurality of convolution layers, and the method comprises:
generating, by a first convolution layer in the plurality of convolution layers, a first intermediate feature map in response to the input feature map and a second intermediate feature map output from a second convolution layer in the plurality of convolution layers; and
performing a splicing processing on the input feature map of the processing unit and intermediate feature map outputs from the plurality of convolution layers to obtain the output feature map of the processing unit.

### Claim 9 (depends on 2)

The method according to claim 2, wherein the second codec comprises an encoding portion of the first codec, a decoding portion of the first codec and a third codec between the encoding portion of the first codec and the decoding portion of the first codec, and target codec comprises an encoding portion of the second codec, a decoding portion of the second codec and a fourth codec between the encoding portion of the second codec and the decoding portion of the second codec.

### Claim 10 (depends on 1)

The method according to claim 1, wherein the acquiring the to-be-processed image comprises:
acquiring the to-be-processed image from a to-be-processed video.

### Claim 11

An apparatus, comprising processing circuitry configured to:
acquire a to-be-processed image;
zoom out the to-be-processed image from a first size to a second size;
perform a splicing processing on a first output feature map from a first processing unit and the to-be-processed image of the second size to obtain a first spliced output feature map for input into a second processing unit of a first codec;
determine a first feature map based on at least a feature reconstruction on the to-be-processed image of the second size by a first codec of a first set of coding units, the first set of coding units comprising at least a pair of encoder and decoder, a coding unit in the first codec including at least the first processing unit and the second processing unit;
zoom in the first feature map to the first size;
perform a splicing processing on the to-be-processed image of the first size and the first feature map of the first size to obtain a spliced feature map; and
perform a feature reconstruction on the spliced feature map by a target codec, to obtain a target image, the target codec comprising a target set of coding units, a first subset of the target set of coding units corresponding to the first set of coding units.

### Claim 12 (depends on 11)

The apparatus according to claim 11, wherein the processing circuitry is configured to:
perform the feature reconstruction on the to-be-processed image of the second size by the first codec to obtain a coarse feature map of the second size;
zoom in the coarse feature map of the second size to obtain a zoomed coarse feature map of a third size between the first size and the second size;
zoom out the to-be-processed image of the first size to the third size;
perform a splicing processing on the zoomed coarse feature map of the third size and the to-be-processed image of the third size to obtain a second feature map; and
perform a feature reconstruction on the second feature map by a second codec, to obtain the first feature map, the second codec comprising a second set of coding units, a second subset of the target set of coding units corresponding to the second set of coding units, and a subset of the second set of coding units corresponding to the first set of coding units.

### Claim 13 (depends on 12)

The apparatus according to claim 12, wherein the processing circuitry is configured to:
combine the zoomed coarse feature map of the third size and the to-be-processed image of the third size in parallel in a channel dimension to obtain the second feature map.

### Claim 14 (depends on 11)

The apparatus according to claim 11, wherein the processing circuitry is configured to:
generate the first output feature map by the first processing unit based on a first input feature map; and
perform a fusion processing on the first output feature map of the first processing unit and the first input feature map of the first processing unit to obtain a first fused feature map for inputting to the second processing unit of the first codec.

### Claim 15 (depends on 11)

The apparatus according to claim 11, wherein the coding unit of the first codec comprises a processing unit that generates an output feature map in response to an input feature map by a plurality of convolution layers, and the processing circuitry is configured to:
generate, by a first convolution layer in the plurality of convolution layers, a first intermediate feature map in response to the input feature map and a second intermediate feature map output from a second convolution layer in the plurality of convolution layers; and
perform a splicing processing on the input feature map of the processing unit and intermediate feature map outputs from the plurality of convolution layers to obtain the output feature map of the processing unit.

### Claim 16 (depends on 12)

The apparatus according to claim 12, wherein the second codec comprises an encoding portion of the first codec, a decoding portion of the first codec and a third codec between the encoding portion of the first codec and the decoding portion of the first codec, and target codec comprises an encoding portion of the second codec, a decoding portion of the second codec and a fourth codec between the encoding portion of the second codec and the decoding portion of the second codec.

### Claim 17 (depends on 11)

The apparatus according to claim 11, wherein the processing circuitry is configured to:
acquire the to-be-processed image from a to-be-processed video.

### Claim 18

A non-transitory computer-readable medium storing instructions which when executed by a computer cause the computer to perform:
acquiring a to-be-processed image;
zooming out the to-be-processed image from a first size to a second size;
performing a splicing processing on a first output feature map from a first processing unit and the to-be-processed image of the second size to obtain a first spliced output feature map for inputting into a second processing unit of a first codec;
determining a first feature map based on at least a feature reconstruction on the to-be-processed image of the second size by a first codec of a first set of coding units, the first set of coding units comprising at least a pair of encoder and decoder, a coding unit in the first codec including at least the first processing unit and the second processing unit;
zooming in the first feature map to the first size;
performing a splicing processing on the to-be-processed image of the first size and the first feature map of the first size to obtain a spliced feature map; and
performing a feature reconstruction on the spliced feature map by a target codec, to obtain a target image, the target codec comprising a target set of coding units, a first subset of the target set of coding units corresponding to the first set of coding units.