# US 11,113,816 B2 · Image segmentation apparatus, method and relevant computing device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,113,816 B2
- Document ID: US11113816B2
- Status: Granted
- Kind: B2
- Application number: 16/651,946
- Filed: 2019-09-19
- Published: 2020-10-22
- Granted: 2021-09-07
- Earliest claimed: 2018-09-30
- Assignees: Boe Technology Group Co., Ltd.
- Applicants: Boe Technology Group Co., Ltd.
- Inventors: Guannan Chen; Lijie Zhang
- Web page: https://patentgrep.com/patent/US11113816B2

## CPC

- G06N 3/08: Learning methods
- G06N 3/02: Neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/11: Region-based segmentation
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06N 3/048: Activation functions
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The present disclosure provides an image segmentation apparatus, method and relevant computing device. The image segmentation apparatus comprises: a feature extractor configured to extract N image semantic features having different scales from an input image, where N is an integer not less than 3; and a feature processor comprising cascaded dense-refine networks and being configured to perform feature processing on the N image semantic features to obtain a binarized mask image for the input image. A dense-refine network is configured to generate a low-frequency semantic feature from semantic features input thereto by performing densely-connected convolution processing on the semantic features respectively to obtain respective image global features, performing feature fusion on the image global features to obtain a fused image global feature, and performing pooling processing on the fused image global feature to generate and output the low-frequency semantic feature. The semantic features are selected from a group consisting of the N image sematic features and low-frequency semantic features generated by dense-refine networks. The feature processor is configured to obtain the binarized mask image based on low-frequency semantic features generated by the dense-refine networks.

## Claims

### Claim 1

An image segmentation apparatus, comprising:
a feature extractor configured to extract N image semantic features having different scales from an input image, where N is an integer not less than 3; and
a feature processor comprising cascaded dense-refine networks and being configured to perform feature processing on the N image semantic features to obtain a binarized mask image of the input image,
wherein each of the dense-refine networks is configured to generate a low-frequency semantic feature from semantic features input thereto by: performing densely-connected convolution processing on the semantic features respectively to obtain respective image global features, performing feature fusion on the image global features to obtain a fused image global feature, and performing pooling processing on the fused image global feature to generate and output the low-frequency semantic feature; and
wherein the semantic features are selected from a group consisting of the N image sematic features and low-frequency semantic features generated by the dense-refine networks, and the feature processor is configured to obtain the binarized mask image based on a low-frequency semantic features generated by the dense-refine networks.

### Claim 2 (depends on 1)

The image segmentation apparatus according to claim 1, wherein each of the dense-refine networks comprises:
L dense convolution units configured to perform densely-connected convolution processing on L semantic features which have different scales and are input to the dense-refine network to obtain respective L image global features, where 1<L<N;
a fusion unit configured to perform feature fusion on the L image global features to obtain a fused image global feature having a specific scale, wherein the specific scale is a maximum scale among scales of the L semantic features; and
a chain pooling unit configured to perform chain pooling processing on the fused image global feature to generate a low-frequency semantic feature having the specific scale;
wherein the feature processor is configured to obtain the binarized mask image based on the low-frequency semantic feature generated by a last dense-refine network in the cascade.

### Claim 3 (depends on 2)

The image segmentation apparatus according to claim 2, wherein the feature processor comprises N−1 dense-refine networks cascaded according to a predetermined cascade relationship, and is configured to perform feature processing on the N image semantic features sequentially in an ascending order of scales;
wherein, given L=2, the predetermined cascade relationship is: two sematic features having adjacent scales and input into a first dense-refine network comprise an image semantic feature having a first scale and an image semantic feature having a second scale, and two sematic features having adjacent scales and input into an nth dense-refine network comprise a low-frequency semantic feature having an nth scale and an image semantic feature having an n+1th scale, and the low-frequency semantic feature having the nth scale is output by an n−1th dense-refine network, where 2≤n≤N−1.

### Claim 4 (depends on 3)

The image segmentation apparatus according to claim 3, wherein each of the dense-refine networks comprises two dense convolution units, and the two dense convolution units are configured to perform densely-connected convolution processing on two sematic features having adjacent scales, respectively, to obtain two image global features having adjacent scales;
the fusion unit is configured to perform feature fusion on the two image global features having adjacent scales to obtain the fused image global feature, and a scale of the fused image global feature is the same as a larger scale of the adjacent scales; and
the chain pooling unit is configured to perform pooling processing on the fused image global feature to generate the low-frequency semantic feature having the larger scale.

### Claim 5 (depends on 2)

The image segmentation apparatus according to claim 2, wherein the dense convolution units comprise M levels of convolution sub-networks, M being an integer not less than two, and the dense convolution units are configured to:
perform convolution operation on the image semantic features through a first-level convolution sub-network to output a respective convolution result; and
perform convolution operation on an input to an mth-level convolution sub-network through the mth-level convolution sub-network to output a respective convolution result, wherein the input is generated by concatenating the image semantic features with the convolution results output by the first to m−1th levels of convolution sub-networks, wherein 2≤m≤M;
wherein an output of an Mt-level convolution sub-network is taken as the image global feature obtained by the dense convolution unit.

### Claim 6 (depends on 2)

The image segmentation apparatus according to claim 2, wherein the chain pooling unit comprises a chain dense pooling unit, the chain dense pooling unit comprises cascaded W pooling networks and a convolutional sub-network, W is an integer not less than 2, and the chain dense pooling unit is configured to:
perform pooling operation on the fused image global feature through a first pooling network to output a respective pooling result;
perform pooling operation on an input to a wth pooling network through the wth pooling network to output a respective pooling result, where the input comprises the pooling result output by a w−1th pooling network, and 1<w<W;
chain-concatenate the fused image global feature with the pooling results output by the W pooling networks to generate a concatenated pooling output; and
perform convolution operation on the concatenated pooling output through the convolution sub-network to generate a respective low-frequency semantic feature.

### Claim 7 (depends on 2)

The image segmentation apparatus according to claim 2, wherein the fusion unit comprises:
convolution sub-units configured to perform convolution operations on the L image global features from the dense convolution units, respectively;
an upsampling subunit configured to upsample image global features which are among convoluted L image global features and have scales smaller than the specific scale; and
a superimposition subunit configured to superimpose the upsampled image global features with the image global feature having the specific scale to generate the fused image global feature having the specific scale.

### Claim 8 (depends on 1)

The image segmentation apparatus according to claim 1, wherein the feature extractor is configured to extract the image semantic features from the input image by using a convolutional neural network VGG-19.

### Claim 9

An image segmentation method, comprising:
extracting N image semantic features having different scales from an input image, where N is an integer not less than 3; and
performing feature processing on the N image semantic features through cascaded dense-refine networks, the feature processing comprising the following steps performed by each of the dense-refine networks:
performing densely-connected convolution processing on semantic features input thereto, respectively, to obtain multiple image global features, the semantic features being selected from a group consisting of the N image sematic features and low-frequency semantic features generated by the dense-refine networks;
performing feature fusion on the multiple image global features to obtain a fused image global feature; and
performing pooling processing on the fused image global feature, to generate and output a low-frequency semantic feature; and
obtaining a binarized mask image of the input image based on a low-frequency semantic feature generated by the define-refine networks.

### Claim 10 (depends on 9)

The image segmentation method according to claim 9, wherein L semantic features having different scales are input to each of the dense-refine networks, and the feature processing comprises the following steps performed by each of the dense-refine networks:
performing densely-connected convolution processing on the L semantic features respectively to obtain L image global features, where 1<L<N;
performing feature fusion on the L image global features to obtain a fused image global feature having a specific scale, wherein the specific scale is a maximum scale among the scales of the L semantic features; and
performing chain pooling processing on the fused image global feature to generate a low-frequency semantic feature having the specific scale;
wherein the binarized mask image is obtained based on a low-frequency semantic feature generated by a last dense-refine network in the cascade.

### Claim 11 (depends on 10)

The image segmentation method according to claim 10, wherein the cascaded dense-refine networks comprise N−1 dense-refine networks cascaded according to a predetermined cascade relationship, and the feature processing is performed on the N image semantic features sequentially in an ascending order of scales,
wherein, given L=2, the predetermined cascade relationship is: two sematic features having adjacent scales and input into a first dense-refine network comprise an image semantic feature having a first scale and an image semantic feature having a second scale, and two sematic features having adjacent scales and input into an nth dense-refine network comprise a low-frequency semantic feature having an nth scale and an image semantic feature having an n+1th scale, and the low-frequency semantic feature having the nth scale is output by an n−1th dense-refine network, where 2≤n≤N−1.

### Claim 12 (depends on 9)

The image segmentation method according to claim 9, wherein the densely-connected convolution processing comprises the following steps performed by M levels of convolution sub-networks:
performing convolution operation on the image semantic features through a first-level convolution sub-network to output a respective convolution result; and
performing convolution operation on an input to an mth-level convolution sub-network through the mth-level convolution sub-network to output a respective convolution result, wherein the input is generated by concatenating the image semantic features with the convolution results output by the first to m−1th levels of convolution sub-networks, wherein 2≤m≤M;
wherein an output of an Mt-level convolution sub-network is taken as a corresponding obtained image global feature.

### Claim 13 (depends on 9)

The image segmentation method according to claim 9, wherein the chain pooling processing comprises the following steps performed by cascaded W pooling networks and a convolution sub-network:
performing pooling operation on the fused image global feature through a first pooling network to output a respective pooling result;
performing pooling operation on an input to a wth pooling network through the wth pooling network to output a respective pooling result, where the input is the pooling result output by a w−1th pooling network, and 1<w<W and W is an integer not less than 2;
concatenating the fused image global feature with the pooling results output by the W pooling networks to generate a concatenated pooling output; and
performing convolution operation on the concatenated pooling output to generate the low-frequency semantic feature.

### Claim 14 (depends on 9)

A computing device, comprising: a processor and a memory;
the memory storing executable instructions, which, when executed by the processor, cause the processor to perform the image segmentation method according to claim 9.

### Claim 15 (depends on 9)

A non-transitory computer storage medium on which computer-executable instructions are stored, the computer-executable instructions, when executed by a processor, causing the processor to perform the image segmentation method according to claim 9.

### Claim 16 (depends on 3)

The image segmentation apparatus according to claim 3, wherein the dense convolution units comprise M levels of convolution sub-networks, M being an integer not less than two, and the dense convolution units are configured to:
perform convolution operation on the image semantic features through a first-level convolution sub-network to output a respective convolution result; and
perform convolution operation on an input to an mth-level convolution sub-network through the mth-level convolution sub-network to output a respective convolution result, wherein the input is generated by concatenating the image semantic features with the convolution results output by the first to m−1th levels of convolution sub-networks, wherein 2≤m≤M;
wherein an output of an Mt-level convolution sub-network is taken as the image global feature obtained by the dense convolution unit.

### Claim 17 (depends on 3)

The image segmentation apparatus according to claim 3, wherein the chain pooling unit comprises a chain dense pooling unit, the chain dense pooling unit comprises cascaded W pooling networks and a convolutional sub-network, W is an integer not less than 2, and the chain dense pooling unit is configured to:
perform pooling operation on the fused image global feature through a first pooling network to output a respective pooling result;
perform pooling operation on an input to a wth pooling network through the wth pooling network to output a respective pooling result, where the input comprises the pooling result output by a w−1th pooling network, and 1<w<W;
chain-concatenate the fused image global feature with the pooling results output by the W pooling networks to generate a concatenated pooling output; and
perform convolution operation on the concatenated pooling output through the convolution sub-network to generate a respective low-frequency semantic feature.

### Claim 18 (depends on 3)

The image segmentation apparatus according to claim 3, wherein the fusion unit comprises:
convolution sub-units configured to perform convolution operations on the L image global features from the dense convolution units, respectively;
an upsampling subunit configured to upsample image global features which are among convoluted L image global features and have scales smaller than the specific scale; and
a superimposition subunit configured to superimpose the upsampled image global features with the image global feature having the specific scale to generate the fused image global feature having the specific scale.

### Claim 19 (depends on 10)

The image segmentation method according to claim 10, wherein the densely-connected convolution processing comprises the following steps performed by M levels of convolution sub-networks:
performing convolution operation on the image semantic features through a first-level convolution sub-network to output a respective convolution result; and
performing convolution operation on an input to an mth-level convolution sub-network through the mth-level convolution sub-network to output a respective convolution result, wherein the input is generated by concatenating the image semantic features with the convolution results output by the first to m−1th levels of convolution sub-networks, wherein 2≤m≤M;
wherein an output of an Mt-level convolution sub-network is taken as a corresponding obtained image global feature.

### Claim 20 (depends on 10)

The image segmentation method according to claim 10, wherein the chain pooling processing comprises the following steps performed by cascaded W pooling networks and a convolution sub-network:
performing pooling operation on the fused image global feature through a first pooling network to output a respective pooling result;
performing pooling operation on an input to a wth pooling network through the wth pooling network to output a respective pooling result, where the input is the pooling result output by a w−1th pooling network, and 1<w<W and W is an integer not less than 2;
concatenating the fused image global feature with the pooling results output by the W pooling networks to generate a concatenated pooling output; and
performing convolution operation on the concatenated pooling output to generate the low-frequency semantic feature.