# US 2021/0278655 A1 · Automated segmentation and guided correction of endothelial cell images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0278655 A1
- Document ID: US20210278655A1
- Status: Published application
- Kind: A1
- Application number: 17/141,564
- Filed: 2021-01-05
- Published: 2021-09-09
- Earliest claimed: 2020-03-05
- Assignees: Case Western Reserve University
- Applicants: Case Western Reserve University
- Inventors: David L. Wilson; Naomi Joseph; Jonathan H. Lass; Beth Ann Benetz; Harry J. Menegay; Elizabeth Fitzpatrick
- Web page: https://patentgrep.com/patent/US20210278655A1

## CPC

- G06T 7/11: Region-based segmentation
- C12N 5/0621: {Eye cells, e.g. cornea, iris pigmented cells (photoreceptors C12N5/062)}
- G02B 21/367: {providing an output produced by processing a plurality of individual source images, e.g. image tiling, montage, composite images, depth sectioning, image comparison}
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/136: involving thresholding
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/695: {Preprocessing, e.g. image segmentation}
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- C12N: MICROORGANISMS OR ENZYMES; COMPOSITIONS THEREOF; PROPAGATING, PRESERVING, OR MAINTAINING MICROORGANISMS; MUTATION OR GENETIC ENGINEERING; CULTURE MEDIA (microbiological testing media C12Q1/00)
- G02B: OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Embodiments discussed herein facilitate automated and/or semi-automated segmentation of endothelial cells via a trained deep learning model. One example embodiment is a method, comprising: accessing an optical microscopy image comprising a set of corneal endothelial cells of a patient of a keratoplasty; pre-processing the optical microscopy image to generate a pre-processed optical microscopy image via correcting for at least one of shading or illumination artifacts in the optical microscopy image; segmenting, based at least in part on a trained deep learning (DL) model, a plurality of corneal endothelial cells of the set of corneal endothelial cells in the pre-processed optical microscopy image; and displaying, via a graphical user interface (GUI), at least the segmented plurality of corneal endothelial cells.

## Claims

### Claim 1

A non-transitory computer-readable medium storing computer-executable instructions that, when executed, cause a processor to perform operations, comprising:
accessing an optical microscopy image comprising a set of corneal endothelial cells of a patient of a keratoplasty;
pre-processing the optical microscopy image to generate a pre-processed optical microscopy image via correcting for at least one of shading or illumination artifacts in the optical microscopy image;
segmenting, based at least in part on a trained deep learning (DL) model, a plurality of corneal endothelial cells of the set of corneal endothelial cells in the pre-processed optical microscopy image; and
displaying, via a graphical user interface (GUI), at least the segmented plurality of corneal endothelial cells.

### Claim 2 (depends on 1)

The non-transitory machine-readable medium of claim 1, wherein the operations further comprise analyzing the segmented plurality of corneal endothelial cells to determine whether one or more cells of the segmented plurality of corneal endothelial cells are potentially under-segmented or potentially over-segmented.

### Claim 3 (depends on 2)

The non-transitory machine-readable medium of claim 2, wherein one or more cells of the segmented plurality of corneal endothelial cells are potentially under-segmented or potentially over-segmented, and wherein the operations further comprise identifying, on the GUI, the one or more cells of the segmented plurality of corneal endothelial cells that are potentially under-segmented or potentially over-segmented.

### Claim 4 (depends on 3)

The non-transitory machine-readable medium of claim 3, wherein the operations further comprise modifying, based on a user input, a segmentation of at least one of the one or more cells of the segmented plurality of corneal endothelial cells that are potentially under-segmented or potentially over-segmented.

### Claim 5 (depends on 3)

The non-transitory machine-readable medium of claim 3, wherein the operations further comprise determining that a first cell of the one or more cells is potentially under-segmented based on the first cell having an area greater than a first threshold.

### Claim 6 (depends on 5)

The non-transitory machine-readable medium of claim 5, wherein the first threshold is three standard deviations greater than a mean cell area of the segmented plurality of corneal endothelial cells.

### Claim 7 (depends on 3)

The non-transitory machine-readable medium of claim 3, wherein the optical microscopy image comprises a plurality of pixels, wherein segmenting the plurality of corneal endothelial cells is based at least in part on a binarization threshold for identifying whether each pixel of the plurality of pixels corresponds to a cell interior or a cell boundary, and wherein the operations further comprise determining that a first cell of the one or more cells is potentially under-segmented based on the first cell corresponding to two or more cells when the segmentation threshold is changed.

### Claim 8 (depends on 3)

The non-transitory machine-readable medium of claim 3, wherein the operations further comprise determining that a first cell of the one or more cells is potentially over-segmented based on the first cell having an area lower than a first threshold.

### Claim 9 (depends on 8)

The non-transitory machine-readable medium of claim 8, wherein the first threshold is three standard deviations lower than a mean cell area of the segmented plurality of corneal endothelial cells.

### Claim 10 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the DL model is a convolutional neural network (CNN).

### Claim 11 (depends on 10)

The non-transitory computer-readable medium of claim 10, wherein the CNN has a U-Net architecture.

### Claim 12 (depends on 1)

The non-transitory computer-readable medium of claim 1, wherein the segmenting comprises performing binarizing a probability map generated by the DL model based on one of an adaptive threshold or an Otsu threshold.

### Claim 13

A non-transitory computer-readable medium storing computer-executable instructions that, when executed, cause a processor to perform operations, comprising:
accessing a training set comprising a plurality of optical microscopy images, wherein each optical microscopy image of the training set comprises an associated set of corneal endothelial cells of a patient associated with that optical microscopy image, and wherein each optical microscopy image of the training set is associated with a ground truth segmentation of plurality of corneal endothelial cells of the set of corneal endothelial cells of that optical microscopy image;
pre-processing each optical microscopy image of the training set to correct for at least one of shading or illumination artifacts; and
based at least on each pre-processed optical microscopy image and the ground truth segmentation for each optical microscopy image, training a deep learning model to segment a plurality of corneal endothelial cells of a set of corneal endothelial cells of an additional pre-processed optical microscopy image.

### Claim 14 (depends on 13)

The non-transitory computer-readable medium of claim 13, wherein the deep learning model is a convolutional neural network (CNN).

### Claim 15 (depends on 14)

The non-transitory computer-readable medium of claim 14, wherein the CNN has a U-Net architecture.

### Claim 16

An apparatus, comprising:
memory configured to store an optical microscopy image comprising a set of corneal endothelial cells of a patient of a keratoplasty;
one or more processors configured to perform operations comprising:
pre-processing the optical microscopy image to generate a pre-processed optical microscopy image via correcting for at least one of shading or illumination artifacts in the optical microscopy image;
segmenting, based at least in part on a trained deep learning (DL) model, a plurality of corneal endothelial cells of the set of corneal endothelial cells in the pre-processed optical microscopy image; and
displaying, via a graphical user interface (GUI), at least the segmented plurality of corneal endothelial cells.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the operations further comprise analyzing the segmented plurality of corneal endothelial cells to determine whether one or more cells of the segmented plurality of corneal endothelial cells are potentially under-segmented or potentially over-segmented.

### Claim 18 (depends on 17)

The apparatus of claim 17, wherein one or more cells of the segmented plurality of corneal endothelial cells are potentially under-segmented or potentially over-segmented, and wherein the operations further comprise identifying, on the GUI, the one or more cells of the segmented plurality of corneal endothelial cells that are potentially under-segmented or potentially over-segmented.

### Claim 19 (depends on 18)

The apparatus of claim 18, wherein the operations further comprise modifying, based on a user input, a segmentation of at least one of the one or more cells of the segmented plurality of corneal endothelial cells that are potentially under-segmented or potentially over-segmented.

### Claim 20 (depends on 18)

The apparatus of claim 18, wherein the operations further comprise determining that a first cell of the one or more cells is potentially under-segmented based on the first cell having an area greater than a first threshold.

### Claim 21 (depends on 20)

The apparatus of claim 20, wherein the first threshold is three standard deviations greater than a mean cell area of the segmented plurality of corneal endothelial cells.

### Claim 22 (depends on 18)

The apparatus of claim 18, wherein the optical microscopy image comprises a plurality of pixels, wherein segmenting the plurality of corneal endothelial cells is based at least in part on a binarization threshold for identifying whether each pixel of the plurality of pixels corresponds to a cell interior or a cell boundary, and wherein the operations further comprise determining that a first cell of the one or more cells is potentially under-segmented based on the first cell corresponding to two or more cells when the segmentation threshold is changed.

### Claim 23 (depends on 18)

The apparatus of claim 18, wherein the operations further comprise determining that a first cell of the one or more cells is potentially over-segmented based on the first cell having an area lower than a first threshold.

### Claim 24 (depends on 23)

The apparatus of claim 23, wherein the first threshold is three standard deviations lower than a mean cell area of the segmented plurality of corneal endothelial cells.
