# US 2023/0410316 A1 · Sequential convolutional neural networks for nuclei segmentation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410316 A1
- Document ID: US20230410316A1
- Status: Published application
- Kind: A1
- Application number: 18/339,193
- Filed: 2023-06-21
- Published: 2023-12-21
- Earliest claimed: 2021-01-12
- Assignees: Provincial Health Services Authroity
- Applicants: Provincial Health Services Authroity
- Inventors: Calum MACAULAY; Paul GALLAGHER
- Web page: https://patentgrep.com/patent/US20230410316A1

## CPC

- G06T 7/11: Region-based segmentation
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/12: Edge-based segmentation
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/82: using neural networks
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06V 20/698: {Matching; Classification}
- G16H 30/40: for processing medical images, e.g. editing
- G06T 2207/10056: Microscopic image
- G06T 2207/20021: Dividing image into blocks, subimages or windows
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Methods and apparatus for segmenting cell nuclei in medical images apply first and second trained machine learning algorithms. The first trained machine learning algorithm processes a medical image to provide center locations of cell nuclei depicted in the image. The second machine learning algorithm processes each of a plurality of patches of the image. Each of the patches correspond to one of the plurality of center locations. Processing each patch yields a nuclear boundary corresponding to the corresponding one of the center locations. The methods and apparatus allow associating individual pixels of the image with one or more than one nuclei and have been shown to be effective for instance segmentation of nuclei in clusters of overlapping cell nuclei.

## Claims

### Claim at position 1

-20. (canceled)

### Claim 21

A method for segmenting cell nuclei in medical images, the method comprising:
by a first trained machine learning algorithm processing a medical image to provide center locations of cell nuclei depicted in the medical image;
by a second trained machine learning algorithm processing each of a plurality of patches of the medical image, each of the patches corresponding to one of the plurality of center locations, the processing by the second trained machine learning algorithm outputting a nuclear boundary corresponding to the corresponding one of the center locations.

### Claim 22 (depends on 21)

The method according to claim 21 wherein the first machine learning algorithm is implemented by a first convolutional neural network.

### Claim 23 (depends on 21)

The method according to claim 21 wherein processing each of the plurality of patches of the medical image by the second machine learning algorithm comprises receiving each of the patches as input to a second convolutional neural network.

### Claim 24 (depends on 23)

The method according to claim 23 wherein the first machine learning algorithm is implemented by a first convolutional neural network and at least one of the first and second convolutional neural networks has a UNet configuration.

### Claim 25 (depends on 21)

The method according to claim 21 wherein the patches of the medical image are centered on the corresponding one of the plurality of center locations.

### Claim 26 (depends on 21)

The method according to claim 21 wherein the patches of a digital histopathology representation have dimension of at least 80 by 80 pixels.

### Claim 27 (depends on 21)

The method according to claim 21 wherein the first machine learning algorithm is implemented by a first convolutional neural network, the second machine learning algorithm is implemented by a second convolutional neural network and the first and second convolutional neural networks have architectures that are one of different from one another and the same as one another.

### Claim 28 (depends on 21)

The method according to claim 21 further comprising obtaining cell information corresponding to the center locations and processing the cell information together with the center locations to perform cell type based cell-cell association quantification.

### Claim 29 (depends on 28)

The method according to claim 28 wherein the cell information comprises morphologically based and/or immunohistochemistry (IHC) based characterization information.

### Claim 30 (depends on 21)

The method according to claim 21 wherein the medical image comprises: a digital histopathology representation, a cytology image, a cytopathology image, or an in vivo histology image.

### Claim 31 (depends on 21)

The method according to claim 21 wherein the medical image includes one or more clusters of overlapping cell nuclei.

### Claim 32 (depends on 21)

The method according to claim 21 comprising applying feature calculations and a binary classification tree to classify objects corresponding to the nuclear boundaries.

### Claim 33

Apparatus for segmenting cell nuclei in medical images, the apparatus comprising:
a first trained machine learning algorithm operative to process a medical image to provide center locations of cell nuclei depicted in the medical image;
a second trained machine learning algorithm operative to process each of a plurality of patches of the medical image, each of the patches corresponding to one of the plurality of center locations, the processing by the second trained machine learning algorithm outputting a nuclear boundary corresponding to the corresponding one of the center locations.

### Claim 34 (depends on 33)

The apparatus according to claim 33 wherein the first machine learning algorithm is implemented by a first convolutional neural network.

### Claim 35 (depends on 33)

The apparatus according to claim 33 wherein the second machine learning algorithm is configured to receive each of the patches as input to a second convolutional neural network.

### Claim 36 (depends on 33)

The apparatus according to claim 33 wherein the patches of the medical image are centered on the corresponding one of the plurality of center locations.

### Claim 37 (depends on 33)

The apparatus according to claim 33 wherein the patches of a digital histopathology representation have dimension of at least 80 by 80 pixels.

### Claim 38 (depends on 33)

The apparatus according to claim 33 wherein the first machine learning algorithm is implemented by a first convolutional neural network, the second machine learning algorithm is implemented by a second convolutional neural network and the first and second convolutional neural networks have architectures that are one of different from one another and the same as one another.

### Claim 39 (depends on 33)

The apparatus according to claim 33 further comprising a data processor configured to obtain cell information corresponding to the center locations and processing the cell information together with the center locations to perform cell type based cell-cell association quantification.

### Claim 40 (depends on 39)

The apparatus according to claim 39 wherein the cell information comprises morphologically based and/or immunohistochemistry (IHC) based characterization information.

### Claim 41 (depends on 33)

The apparatus according to claim 33 wherein the medical image comprises: a digital histopathology representation, a cytology image, a cytopathology image, or an in vivo histology image.

### Claim 42 (depends on 33)

The apparatus according to claim 33 wherein the apparatus is operable to instance segment individual cell nuclei in one or more clusters of overlapping cell nuclei included in the medical image.

### Claim 43 (depends on 33)

The apparatus according to claim 33 comprising a data processor configured to apply one or more feature calculations and a binary classification tree to classify objects corresponding to the nuclear boundaries.
