# US 12,008,797 B2 · Image segmentation method and image processing apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,797 B2
- Document ID: US12008797B2
- Status: Granted
- Kind: B2
- Application number: 17/383,181
- Filed: 2021-07-22
- Published: 2021-11-11
- Granted: 2024-06-11
- Earliest claimed: 2019-03-01
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Zhi Tian; Tong He; Chunhua Shen; Youliang Yan; Songcen Xu; Yiren Zhou; Xiaofei Wu; Jianzhuang Liu
- Web page: https://patentgrep.com/patent/US12008797B2

## CPC

- G06V 10/457: {by analysing connectivity, e.g. edge linking, connected component analysis or slices}
- G06F 18/2135: based on approximation criteria, e.g. principal component analysis
- G06F 18/2137: based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/253: {of extracted features}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/7715: {Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods}
- G06V 10/82: using neural networks
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

This application discloses an image segmentation method in the field of artificial intelligence. The method includes: obtaining an input image and a processing requirement; performing multi-layer feature extraction on the input image to obtain a plurality of feature maps; downsampling the plurality of feature maps to obtain a plurality of feature maps with a reference resolution, where the reference resolution is less than a resolution of the input image; fusing the plurality of feature maps with the reference resolution to obtain at least one feature map group; upsampling the feature map group by using a transformation matrix W, to obtain a target feature map group; and performing target processing on the target feature map group based on the processing requirement to obtain a target image.

## Claims

### Claim 1

An image segmentation method, comprising:
obtaining an input image and a processing requirement, wherein the processing requirement is used to indicate to perform target processing on a target feature map group obtained by performing image segmentation on the input image;
performing multi-layer feature extraction on the input image to obtain a plurality of feature maps;
downsampling the plurality of feature maps to obtain a plurality of feature maps with a reference resolution, wherein the reference resolution is less than a resolution of the input image;
fusing the plurality of feature maps with the reference resolution to obtain at least one feature map group;
upsampling the at least one feature map group by using a transformation matrix W, to obtain the target feature map group, wherein the target feature map group has a same resolution as that of the input image, the transformation matrix W is obtained by modeling training data of an image segmentation task, and one dimension of the transformation matrix W is the same as a quantity of channels of the feature group; and
performing the target processing on the target feature map group based on the processing requirement to obtain a target image.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the upsampling the at least one feature map group by using the transformation matrix W, to obtain the target feature map group comprises:
calculating a product of the transformation matrix W and each of (H×W) one-dimensional matrices that each comprise C elements, to obtain (H×W) one-dimensional matrices that each comprise P elements, wherein an element comprised in any one of the (H×W) one-dimensional matrices that each comprise C elements is an element at a same location in each of C two-dimensional (H×W) matrices comprised in the feature map group, H and W are two dimensions of the feature map group, C is the quantity of channels of the feature map group, the transformation matrix is a two-dimensional (C×P) matrix obtained based on M annotated images comprised in the training data, P=A×B×N, and N is a quantity of categories into which image semantics in the M annotated images are segmented; and
separately performing feature permutation on the (H×W) one-dimensional matrices that each comprise P elements, to obtain the target feature map group, wherein at least one (A×B×N) submatrix comprised in the target feature map group is obtained based on one of the (H×W) one-dimensional matrices that each comprise P elements, and H, W, C, N, P, M, A, and B are all integers greater than 0.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the separately performing feature permutation on the (H×W) one-dimensional matrices that each comprise P elements, to obtain the target feature map group comprises:
determining, based on any one of the (H×W) one-dimensional matrices that each comprise P elements, (A×B) one-dimensional matrices that each comprise N elements; and
using, as a submatrix comprised in the target feature map group, a three-dimensional (A×B×N) matrix obtained based on the (A×B) one-dimensional matrices that each comprise N elements.

### Claim 4 (depends on 2)

The method according to claim 2, wherein any one of the M annotated images is a three-dimensional (H×W×N) matrix, and the transformation matrix W is obtained by performing the following operations:
obtaining at least one (A×B×N) submatrix corresponding to each of the M annotated images to obtain a plurality of (A×B×N) submatrices;
obtaining, based on the plurality of (A×B×N) submatrices, a plurality of vectors comprising P elements, wherein a vector comprising P elements is obtained based on each of the plurality of (A×B×N) submatrices;
performing principal component analysis on the plurality of vectors comprising P elements to obtain a two-dimensional (P×P) matrix; and
using one (C×P) submatrix comprised in the two-dimensional (P×P) matrix as the transformation matrix W.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the performing multi-layer feature extraction on the input image to obtain the plurality of feature maps comprises:
performing a convolution operation on the input image to obtain a first feature map, and performing a convolution operation on a (K−1)th feature map to obtain a Kth feature map, wherein the Kth feature map is a feature map with the reference resolution, a resolution of the (K−1)th feature map is not greater than that of the Kth feature map, K is an integer greater than 1, and the plurality of feature maps comprise K feature maps; and
wherein the downsampling the plurality of feature maps to obtain the plurality of feature maps with the reference resolution comprises:
downsampling the first feature map to obtain a feature map with the reference resolution, and downsampling the (K−1)th feature map to obtain a feature map with the reference resolution.

### Claim 6 (depends on 2)

The method according to claim 2, wherein the fusing the plurality of feature maps with the reference resolution to obtain the at least one feature map group comprises:
stitching the plurality of feature maps with the reference resolution in a channel dimension to obtain the at least one feature map group, wherein the at least one feature map group is a three-dimensional (H×W×C) matrix and corresponds to the C two-dimensional (H×W) matrices; and
wherein the calculating the product of the transformation matrix W and each of the (H×W) one-dimensional matrices that each comprise the C elements to obtain the (H×W) one-dimensional matrices that each comprise the P elements comprises:
calculating a product of the transformation matrix and a one-dimensional matrix corresponding to each element location in the feature map group, to obtain the (H×W) one-dimensional matrices that each comprise P elements, wherein an element comprised in a one-dimensional matrix corresponding to one element location in the feature map group is an element at a same element location in each of the C two-dimensional (H×W) matrices.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising:
obtaining the transformation matrix W;
processing a training sample by using a convolutional neural network, to obtain an image segmentation result of the training sample, wherein the training sample is comprised in the training data;
determining, based on the image segmentation result of the training sample and a standard result corresponding to the training sample, a loss corresponding to the training sample, wherein the standard result is a result expected to be obtained by processing the training sample by using the convolutional neural network; and
updating a parameter of the convolutional neural network by using an optimization algorithm and the loss corresponding to the training sample;
wherein the performing multi-layer feature extraction on the input image to obtain the plurality of feature maps comprises:
inputting the input image into the convolutional neural network and performing the multi-layer feature extraction, to obtain the plurality of feature maps.

### Claim 8

An image processing apparatus, comprising:
a processor; and
a memory storing instructions that when executed by the processor configure the image processing apparatus to:
obtain an input image and a processing requirement, wherein the processing requirement is used to indicate to perform target processing on a target feature map group obtained by performing image segmentation on the input image; and
perform multi-layer feature extraction on the input image to obtain a plurality of feature maps;
downsample the plurality of feature maps to obtain a plurality of feature maps with a reference resolution, wherein the reference resolution is less than a resolution of the input image;
fuse the plurality of feature maps with the reference resolution to obtain at least one feature map group;
upsample the at least one feature map group by using a transformation matrix W, to obtain the target feature map group, wherein the target feature map group has a same resolution as that of the input image, the transformation matrix W is obtained by modeling training data of an image segmentation task, and one dimension of the transformation matrix W is the same as a quantity of channels of the feature group; and
perform the target processing on the target feature map group based on the processing requirement to obtain a target image.

### Claim 9 (depends on 8)

The apparatus according to claim 8, wherein when the image processing apparatus is configured to upsample the at least one feature map group by using the transformation matrix W, to obtain the target feature map group, the image processing apparatus is specifically configured to:
calculate a product of the transformation matrix W and each of (H×W) one-dimensional matrices that each comprise C elements, to obtain (H×W) one-dimensional matrices that each comprise P elements, wherein an element comprised in any one of the (H×W) one-dimensional matrices that each comprise C elements is an element at a same location in each of C two-dimensional (H×W) matrices comprised in the feature map group, H and W are two dimensions of the feature map group, C is the quantity of channels of the feature map group, the transformation matrix is a two-dimensional (C×P) matrix obtained based on M annotated images comprised in the training data, P=A×B×N, and N is a quantity of categories into which image semantics in the M annotated images are segmented; and
separately perform feature permutation on the (H×W) one-dimensional matrices that each comprise P elements, to obtain the target feature map group, wherein at least one (A×B×N) submatrix comprised in the target feature map group is obtained based on one of the (H×W) one-dimensional matrices that each comprise P elements, and H, W, C, N, P, M, A, and B are all integers greater than 0.

### Claim 10 (depends on 8)

The apparatus according to claim 8, wherein when the image processing apparatus is configured to separately perform feature permutation on the (H×W) one-dimensional matrices that each comprise the P elements, to obtain the target feature map group, the image processing apparatus is specifically configured to:
determine, based on any one of the (H×W) one-dimensional matrices that each comprise P elements, (A×B) one-dimensional matrices that each comprise N elements; and
use, as a submatrix comprised in the target feature map group, a three-dimensional (A×B×N) matrix obtained based on the (A×B) one-dimensional matrices that each comprise N elements.

### Claim 11 (depends on 9)

The apparatus according to claim 9, wherein any one of the M annotated images is a three-dimensional (H×W×N) matrix; and
wherein the transformation matrix W is obtained by the image processing apparatus being further configured to:
obtain at least one (A×B×N) submatrix corresponding to each of the M annotated images, to obtain a plurality of (A×B×N) submatrices;
obtain, based on the plurality of (A×B×N) submatrices, a plurality of vectors comprising P elements, wherein a vector comprising P elements is obtained based on each of the plurality of (A×B×N) submatrices;
perform principal component analysis on the plurality of vectors comprising P elements to obtain a two-dimensional (P×P) matrix; and
use one (C×P) submatrix comprised in the two-dimensional (P×P) matrix as the transformation matrix W.

### Claim 12 (depends on 8)

The apparatus according to claim 8, wherein when the image processing apparatus is configured to perform multi-layer feature extraction on the input image to obtain the plurality of feature maps, the image processing apparatus is specifically configured to:
perform a convolution operation on the input image to obtain a first feature map, and perform a convolution operation on a (K−1)th feature map to obtain a Kth feature map, wherein the Kth feature map is a feature map with the reference resolution, a resolution of the (K−1)th feature map is not greater than that of the Kth feature map, K is an integer greater than 1, and the plurality of feature maps comprise K feature maps; and
wherein when the image processing apparatus is configured to downsample the plurality of feature maps to obtain the plurality of feature maps with the reference resolution, the image processing apparatus is specifically configured to:
downsample the first feature map to obtain a feature map with the reference resolution, and downsample the (K−1)th feature map to obtain a feature map with the reference resolution.

### Claim 13 (depends on 9)

The apparatus according to claim 9, wherein when the image processing apparatus is configured to fuse the plurality of feature maps with the reference resolution to obtain the at least one feature map group, the image processing apparatus is specifically configured to:
stitch the plurality of feature maps with the reference resolution in a channel dimension to obtain the at least one feature map group, wherein the feature map group is a three-dimensional (H×W×C) matrix and corresponds to the C two-dimensional (H×W) matrices; and
wherein when the image processing apparatus is configured to calculate the product of the transformation matrix W and each of the (H×W) one-dimensional matrices that each comprise the C elements, to obtain the (H×W) one-dimensional matrices that each comprise the P elements, the image processing apparatus is specifically configured to:
calculate a product of the transformation matrix and a one-dimensional matrix corresponding to each element location in the feature map group, to obtain the (H×W) one-dimensional matrices that each comprise P elements, wherein an element comprised in a one-dimensional matrix corresponding to one element location in the feature map group is an element at a same element location in each of the C two-dimensional (H×W) matrices.

### Claim 14 (depends on 8)

The apparatus according to claim 8, wherein the image processing apparatus is further configured to:
obtain the transformation matrix W;
process a training sample by using a convolutional neural network, to obtain an image segmentation result of the training sample, wherein the training sample is comprised in the training data;
determine, based on the image segmentation result of the training sample and a standard result corresponding to the training sample, a loss corresponding to the training sample, wherein the standard result is a result expected to be obtained by processing the training sample by using the convolutional neural network; and
update a parameter of the convolutional neural network by using an optimization algorithm and the loss corresponding to the training sample; and
wherein when the image processing apparatus is configured to perform multi-layer feature extraction on the input image to obtain the plurality of feature maps, the image processing apparatus is specifically configured to:
input the input image into the convolutional neural network and perform the multi-layer feature extraction, to obtain the plurality of feature maps.

### Claim 15

A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores a computer program, the computer program comprises a program instruction, and when the program instruction is executed by a processor, the processor is configured to perform steps comprising:
obtaining an input image and a processing requirement, wherein the processing requirement is used to indicate to perform target processing on a target feature map group obtained by performing image segmentation on the input image;
performing multi-layer feature extraction on the input image to obtain a plurality of feature maps;
downsampling the plurality of feature maps to obtain a plurality of feature maps with a reference resolution, wherein the reference resolution is less than a resolution of the input image;
fusing the plurality of feature maps with the reference resolution to obtain at least one feature map group;
upsampling the at least one feature map group by using a transformation matrix W, to obtain the target feature map group, wherein the target feature map group has a same resolution as that of the input image, the transformation matrix W is obtained by modeling training data of an image segmentation task, and one dimension of the transformation matrix W is the same as a quantity of channels of the feature group; and
performing the target processing on the target feature map group based on the processing requirement to obtain a target image.