# US 2021/0241027 A1 · Image segmentation method and apparatus, diagnosis system, storage medium, and computer device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241027 A1
- Document ID: US20210241027A1
- Status: Published application
- Kind: A1
- Application number: 17/204,894
- Filed: 2021-03-17
- Published: 2021-08-05
- Earliest claimed: 2018-11-30
- Assignees: Tencent Technology (Shenzhen) Company Limited
- Applicants: Tencent Technology (Shenzhen) Company Limited
- Inventors: Yifan HU; Yefeng ZHENG
- Web page: https://patentgrep.com/patent/US20210241027A1

## CPC

- G06T 7/11: Region-based segmentation
- G06F 18/253: {of extracted features}
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 7/0012: {Biomedical image inspection}
- G06V 10/806: {of extracted features}
- G06V 30/2504: {Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches}
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30016: Brain
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/031: of internal organs
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

An image segmentation method is provided for a computing device. The method includes obtaining a general tumor image, performing tumor localization on the tumor image to obtain a candidate image indicating a position of a tumor region in the general tumor image, inputting the candidate image to a cascaded segmentation network constructed based on a machine learning model, and performing image segmentation on the general tumor region in the candidate image using a first-level segmentation network and a second-level segmentation network in the cascaded segmentation network to obtain a segmented image.

## Claims

### Claim 1

An image segmentation method, performed by a computing device, the method comprising:
obtaining a general tumor image;
performing tumor localization on the tumor image to obtain a candidate image indicating a position of a tumor region in the general tumor image;
inputting the candidate image to a cascaded segmentation network constructed based on a machine learning model; and
performing image segmentation on the general tumor region in the candidate image using a first-level segmentation network and a second-level segmentation network in the cascaded segmentation network to obtain a segmented image.

### Claim 2 (depends on 1)

The method according to claim 1, wherein the cascaded segmentation network further includes a third-level segmentation network, and wherein performing image segmentation on the tumor region in the candidate image using the first-level segmentation network and the second-level segmentation network in the cascaded segmentation network to obtain the segmented image comprises:
performing image segmentation on the candidate image using the first-level segmentation network to obtain a first-level intermediate segmented image marked with the general tumor region;
performing image segmentation on the first-level intermediate segmented image using the second-level segmentation network to obtain a second-level intermediate segmented image marked with the general tumor region and a tumor core region; and
performing image segmentation on the second-level intermediate segmented image using the third-level segmentation network to obtain the segmented image marked with the general tumor region, the tumor core region, and an enhancing tumor core region.

### Claim 3 (depends on 2)

The method according to claim 2, wherein an input image is the candidate image, the first-level intermediate segmented image, or the second-level intermediate segmented image, wherein an output image is the first-level intermediate segmented image, the second-level intermediate segmented image, or the segmented image, wherein the segmentation network includes an upsampling stage and a downsampling stage, and wherein performing image segmentation on the input image by using the segmentation network comprises:
extracting key features from the input image in the downsampling stage of the segmentation network;
performing multi-scale feature fusion on the key features in the upsampling stage of the segmentation network to obtain a feature map;
performing class prediction on pixel points in the feature map to obtain classes of the pixel points in the feature map; and
marking pixel points of a designated class in the feature map to obtain the output image.

### Claim 4 (depends on 3)

The method according to claim 3, wherein extracting key features from the input image in the downsampling stage of the segmentation network comprises:
inputting the input image to the downsampling stage of the segmentation network and performing convolution and downsampling using several first basic network layers to obtain intermediate features; and
performing convolution on the intermediate features using the several first dense block layers to obtain the key features.

### Claim 5 (depends on 4)

The method according to claim 4, wherein the first basic network layer includes a second dense block layer and a pooling layer, and wherein inputting the input image to the downsampling stage of the segmentation network and performing convolution and downsampling by using the several first basic network layers to obtain intermediate features comprises:
inputting the input image to a first first basic network layer in the downsampling stage of the segmentation network, and performing convolution on the input image by using a second dense block layer in the first basic network layer;
performing, by using a pooling layer in the first first basic network layer, downsampling on features outputted after the convolution, to output the downsampled features to a second first basic network layer; and
traversing the other first basic network layers in the several first basic network layers according to a feature propagation sequence, and after the traversal is performed, using features obtained after downsampling is performed by a last first basic network layer as the intermediate features.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the upsampling stage and the downsampling stage are symmetric to each other, the upsampling stage comprises several third dense block layers and several second basic network layers connected sequentially, and the second basic network layer comprises an upsampling layer and a fourth dense block layer connected sequentially, and wherein inputting the key features to the upsampling stage of the segmentation network and performing multi-scale feature fusion to obtain the feature map comprises:
inputting the key features to the upsampling stage of the segmentation network, and performing deconvolution by using the several third dense block layers, to obtain first scale features, to input the first scale features to first several second basic network layers;
performing, in the first several second basic network layers, feature fusion of fourth dense block layers in the first several second basic network layers and second dense block layers in several of the first basic network layers that are symmetric to each other;
performing, in a last second basic network layer by using an upsampling layer in the last second basic network layer, upsampling on features outputted after the fusion in the first several second basic network layers, to obtain second scale features;
obtaining features outputted after convolution of the second dense block layers in the several first basic network layers, and using the obtained features as third scale features; and
performing fusion on the first scale features, the second scale features, and the third scale features by using a fourth dense block layer in the last second basic network layer, and performing deconvolution, to obtain a feature map corresponding to the input image.

### Claim 7 (depends on 6)

The method according to claim 6, wherein performing, in the first several second basic network layers, feature fusion of fourth dense block layers in the first several second basic network layers and second dense block layers in several of the first basic network layers that are symmetric to each other comprises:
inputting the first scale features to a first second basic network layer, and performing upsampling on the inputted first scale features by using an upsampling layer in the first second basic network layer;
obtaining, based on a fourth dense block layer in the first second basic network layer, features outputted after convolution of a second dense block layer in the several first basic network layers that is symmetric to the fourth dense block layer, and combining the obtained features with the upsampled features, to obtain combined features;
performing deconvolution on the combined features by using the fourth dense block layer in the first second basic network layer, to output the deconvoluted features to the second second basic network layer; and
traversing the other second basic network layers in the first several second basic network layers according to a feature propagation sequence, and after the traversal is performed, performing feature fusion of the fourth dense block layers and the second dense block layers that are symmetric to each other.

### Claim 8 (depends on 6)

The method according to claim 6, wherein each convolution layer in the first dense block layers and the third dense block layers comprises several three-dimensional convolution kernels; and
each convolution layers in the second dense block layers and the fourth dense block layers comprises several tangent convolution kernels and several normal convolution kernels.

### Claim 9 (depends on 1)

The method according to claim 1, further comprising:
constructing the cascaded segmentation network based on the machine learning model, wherein the machine learning model is a convolutional neural network model.

### Claim 10 (depends on 9)

The method according to claim 9, wherein constructing the cascaded segmentation network based on the machine learning model comprises:
obtaining training samples carrying labels, wherein the training samples are tumor images on which the general tumor region, the tumor core region, and the enhancing tumor core region are labeled with different types of labels;
establishing a plurality of training sample sets according to types of the labels carried by the training samples, each of the training sample sets corresponding to a type;
performing model training on a plurality of convolutional neural network models having designated model structures respectively by using the plurality of training sample sets; and
cascading the plurality of convolutional neural network models after the model training is performed to obtain the cascaded segmentation network.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the performing tumor localization on the tumor image to obtain the candidate image comprises:
extracting a feature map from the tumor image based on a three-dimensional U-shaped fully convolutional neural network;
performing class prediction on pixel points in the feature map corresponding to the tumor image to obtain classes of the pixel points in the feature map; and
obtaining the candidate image on which the general tumor region is accommodated in a designated region according to pixel points belonging to a general tumor region class in the feature map.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the three-dimensional U-shaped fully convolutional neural network includes an encoder network and a decoder network, and wherein extracting the feature map from the tumor image comprises:
extracting context features of the tumor image using the encoder network; and
extracting localization features of the general tumor region using the decoder network, and performing feature fusion of the context features and the localization features to obtain the feature map.

### Claim 13 (depends on 12)

The method according to claim 12, wherein extracting localization features of the general tumor region using the decoder network and performing feature fusion of the context features and the localization features to obtain the feature map comprises:
performing upsampling on localization features corresponding to a deepest upsampling layer in the decoder network, to obtain to-be-fused features, wherein the localization features corresponding to the deepest upsampling layer are context features corresponding to a deepest downsampling layer in the encoder network;
inputting the to-be-fused features to a second deepest upsampling layer, combining the to-be-fused features with context features corresponding to a second deepest downsampling layer, and obtaining localization features corresponding to the second deepest upsampling layer by performing deconvolution;
traversing the other upsampling layers in descending order of depth, to obtain localization features corresponding to the traversed upsampling layers; and
obtaining the feature map according to localization features corresponding to a shallowest upsampling layer after the traversal is performed.

### Claim 14

An image segmentation apparatus, comprising: a memory storing computer program instructions; and a processor coupled to the memory and configured to executing the computer program instructions and perform:
obtaining a general tumor image;
performing tumor localization on the tumor image to obtain a candidate image indicating a position of a tumor region in the general tumor image;
inputting the candidate image to a cascaded segmentation network constructed based on a machine learning model; and
performing image segmentation on the general tumor region in the candidate image using a first-level segmentation network and a second-level segmentation network in the cascaded segmentation network to obtain a segmented image.

### Claim 15 (depends on 14)

The image segmentation apparatus of claim 14, wherein the cascaded segmentation network further includes a third-level segmentation network, and wherein performing image segmentation on the tumor region in the candidate image using the first-level segmentation network and the second-level segmentation network in the cascaded segmentation network to obtain the segmented image comprises:
performing image segmentation on the candidate image using the first-level segmentation network to obtain a first-level intermediate segmented image marked with the general tumor region;
performing image segmentation on the first-level intermediate segmented image using the second-level segmentation network to obtain a second-level intermediate segmented image marked with the general tumor region and a tumor core region; and
performing image segmentation on the second-level intermediate segmented image using the third-level segmentation network to obtain the segmented image marked with the general tumor region, the tumor core region, and an enhancing tumor core region.

### Claim 16 (depends on 15)

The image segmentation apparatus of claim 15, wherein an input image is the candidate image, the first-level intermediate segmented image, or the second-level intermediate segmented image, wherein an output image is the first-level intermediate segmented image, the second-level intermediate segmented image, or the segmented image, wherein the segmentation network includes an upsampling stage and a downsampling stage, and wherein performing image segmentation on the input image by using the segmentation network comprises:
extracting key features from the input image in the downsampling stage of the segmentation network;
performing multi-scale feature fusion on the key features in the upsampling stage of the segmentation network to obtain a feature map;
performing class prediction on pixel points in the feature map to obtain classes of the pixel points in the feature map; and
marking pixel points of a designated class in the feature map to obtain the output image.

### Claim 17 (depends on 16)

The image segmentation apparatus of claim 16, wherein extracting key features from the input image in the downsampling stage of the segmentation network comprises:
inputting the input image to the downsampling stage of the segmentation network and performing convolution and downsampling using several first basic network layers to obtain intermediate features; and
performing convolution on the intermediate features using the several first dense block layers, to obtain the key features.

### Claim 18 (depends on 17)

The image segmentation apparatus of claim 17, wherein the first basic network layer includes a second dense block layer and a pooling layer, and wherein inputting the input image to the downsampling stage of the segmentation network and performing convolution and downsampling by using the several first basic network layers to obtain intermediate features comprises:
inputting the input image to a first first basic network layer in the downsampling stage of the segmentation network, and performing convolution on the input image by using a second dense block layer in the first basic network layer;
performing, by using a pooling layer in the first first basic network layer, downsampling on features outputted after the convolution, to output the downsampled features to a second first basic network layer; and
traversing the other first basic network layers in the several first basic network layers according to a feature propagation sequence, and after the traversal is performed, using features obtained after downsampling is performed by a last first basic network layer as the intermediate features.

### Claim 19 (depends on 18)

The image segmentation apparatus of claim 18, wherein the upsampling stage and the downsampling stage are symmetric to each other, the upsampling stage comprises several third dense block layers and several second basic network layers connected sequentially, and the second basic network layer comprises an upsampling layer and a fourth dense block layer connected sequentially, and wherein inputting the key features to the upsampling stage of the segmentation network and performing multi-scale feature fusion to obtain the feature map comprises:
inputting the key features to the upsampling stage of the segmentation network, and performing deconvolution by using the several third dense block layers, to obtain first scale features, to input the first scale features to first several second basic network layers;
performing, in the first several second basic network layers, feature fusion of fourth dense block layers in the first several second basic network layers and second dense block layers in several of the first basic network layers that are symmetric to each other;
performing, in a last second basic network layer by using an upsampling layer in the last second basic network layer, upsampling on features outputted after the fusion in the first several second basic network layers, to obtain second scale features;
obtaining features outputted after convolution of the second dense block layers in the several first basic network layers, and using the obtained features as third scale features; and
performing fusion on the first scale features, the second scale features, and the third scale features by using a fourth dense block layer in the last second basic network layer, and performing deconvolution, to obtain a feature map corresponding to the input image.

### Claim 20

A non-transitory computer-readable storage medium storing computer program instructions executable by at least one processor to perform:
obtaining a general tumor image;
performing tumor localization on the tumor image to obtain a candidate image indicating a position of a tumor region in the general tumor image;
inputting the candidate image to a cascaded segmentation network constructed based on a machine learning model; and
performing image segmentation on the general tumor region in the candidate image using a first-level segmentation network and a second-level segmentation network in the cascaded segmentation network to obtain a segmented image.