# US 12,002,206 B2 · System and method for automatically identifying mitosis in H and E stained breast cancer pathological images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,002,206 B2
- Document ID: US12002206B2
- Status: Granted
- Kind: B2
- Application number: 18/332,302
- Filed: 2023-06-09
- Published: 2023-12-14
- Granted: 2024-06-04
- Earliest claimed: 2022-06-12
- Assignees: Guilin University of Electronic Technology
- Applicants: Guilin University of Electronic Technology
- Inventors: Xipeng Pan; Xinjun Bian; Yinghua Lu; Zhenbing Liu; Zujun Qin; Rushi Lan; Huihua Yang; Huadeng Wang; Lingqiao Li; Zimin Wang; Jijun Cheng; Zhizhen Wang; Zhengyun Feng; Shilong Song
- Web page: https://patentgrep.com/patent/US12002206B2

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06V 20/698: {Matching; Classification}
- G06T 2207/10056: Microscopic image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20132: Image cropping
- G06T 2207/30068: Mammography; Breast
- G06T 2207/30096: Tumor; Lesion
- G06T 2207/30204: Marker
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A system and a method for automatically identifying mitosis in H&E stained breast cancer pathological images are provided, belonging to the technical field of digital image processing, and including an input image preprocessing module: cutting an original picture according to a predetermined patch size, and performing a data enhancement by means of picture flipping, rotation, and the like; and a segmentation module: training a segmentation network by cutting patches in a training set, cutting data of a test set according to a corresponding size and sending to the segmentation network to obtain a patch-level segmentation result, and then reconstructing a segmented result into an image belonging to an original size according to patch coordinate information intercepted in a preprocessing stage of it.

## Claims

### Claim 1

A system for automatically identifying mitosis in hematoxylin-eosin (H&E) stained breast cancer pathological images, comprising:
an input image preprocessing module: cutting an original picture into a predetermined patch size, and performing a data enhancement through picture flipping and rotation;
the original picture obtains a mask binary image through a pixel position labeling;
obtaining training data of a segmentation network using a data set labeled at a pixel level, obtaining a corresponding mask binary image through the pixel position labeling, and cutting a corresponding RGB image based on the mask binary image to obtain mitotic patches and non-mitotic patches as the training data;
a segmentation module: training the segmentation network by cutting mitotic patches and non-mitotic patches from a training set, cutting data of a test set according to a corresponding size and sending it to the segmentation network to obtain a patch-level segmentation result, and then reconstructing a segmented result into an image of the original size according to patch coordinate information intercepted in a preprocessing stage of it the segmentation module;
a classification module: training three classification networks, performing a decision-making level fusion on results of the three classification networks, using a voting method to obtain a final output result, and then restoring and marking the final output result to obtain a final detection result;
the segmentation module comprises an attention segmentation network module and a candidate cell selection method module;
the classification module comprises a three-branch classification network module, a decision-making fusion algorithm module and a classification result marking module; and
the attention segmentation network module introduces an SE (Squeeze-and-Excitation) module:
constructing a segmentation network based on Attention U-Net, combining an attention mechanism to realize a weighted extraction of different features in spaces and channels, aiming at morphological features of research objectives mitotic cells, improving a squeeze operation in the SE module, and taking into account both texture information and background information; and based on above characteristics, enabling the segmentation network SEAttUnet based on the Attention U-Net to realize a redistribution of attentions of space positions in different channels and feature maps.

### Claim 2 (depends on 1)

A method of the system for automatically identifying mitosis in H&E stained breast cancer pathological images according to claim 1, comprising following specific steps:
S1: data preprocessing: performing a cutting operation on the training data, wherein an edge length after cutting is 64, and a size of an original high power field (HPF) is not capable of guaranteeing a complete cutting without overlapping, a way of filling first and then cutting is adopted; filling a corresponding size in an outer circle of an HPF edge; before cutting, according to an edge length a of the original HPF and an edge length b of the patch after cutting, calculating a size c to be filled, copying long blocks with a length× a width of a× c in an edge area and filling into the edge area, and filling blocks of a small square area in a lower right corner, selecting a same size area of c×c in a lower right corner of the original HPF for copying; after above operations, obtaining an HPF image completely cut without overlapping, and then cutting all test sets in sequence from an upper left corner to obtain all patches for testing;
S2: a segmentation stage:
through the segmentation network, obtaining a binary mask segmentation result with a patch as a unit after segmentation; restoring all patch-level pictures of a segmentation result to an HPF level according to previously recorded coordinate information, then through operations of connected domain screening and corrosion expansion, filtering out some noise, and obtaining a segmentation binary image of the HPF level;
in the HPF image obtained in a previous step, outline and position information of all mitotic candidate cells is capable to be obtained, using the obtained information of the mitotic candidate cells, center coordinates of the patch are calculated, and a corresponding patch is intercepted in an original image as the candidate cells for the classification; and
S3: a classification stage: training the three classification networks for the classification, using VGG11, VGG19 and ResNet-50 for training respectively; during testing, each of the networks outputs a test result respectively, and prediction results of different classifiers in the same data set are not exactly same; the same patch to be classified as class 0 based on the judgment of a network 1, and potentially as class 1 when assessed by network 2 in a second stage of the classification, three different classification networks are used to synthesize all results for outputting; by assigning different weights to prediction results of each of the networks, different classifiers are capable of learning different features; at a same time, in processes of repeated training and testing, the different classifiers have different judgments on a same cell, according to classification precisions of different networks, the different weights are assigned to adjust contributions of the each of the networks when outputting a final classification result;
fusing classification results, that is, comprehensively considering categories output and classified already in the previous step at a result level, fusing the results of the three networks at the decision-making level in the classification module, and using the voting method to obtain the final output result; and
after passing the classification networks, enabling most of wrongly detected patches to be filtered out with remaining patches as the final result; and restoring and marking the final result to obtain the final detection result.