# US 2017/0372471 A1 · Method and system for detecting pathological anomalies in a digital pathology image and method for annotating a tissue slide

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2017/0372471 A1
- Document ID: US20170372471A1
- Status: Published application
- Kind: A1
- Application number: 15/195,526
- Filed: 2016-06-28
- Published: 2017-12-28
- Earliest claimed: 2016-06-28
- Assignees: Contextvision AB
- Applicants: ContextVision AB
- Inventors: Kristian EURÈN
- Web page: https://patentgrep.com/patent/US20170372471A1

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/2431: Multiple classes
- G06V 20/698: {Matching; Classification}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/20132: Image cropping
- G06T 2207/30024: Cell structures in vitro; Tissue sections in vitro
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for annotating a tissue slide image, a system and a method performed by a computing system for detecting pathological anomalies in a digital pathology image are disclosed. The method performed by a computing system for detecting pathological anomalies in a digital pathology image includes providing a digital pathology image to the computing system and analyzing the digital pathology image using an identification module arranged on the computing system. The identification module uses a machine learning module to execute recognizing an object containing an abnormal image pattern using an identification model loaded in said identification module and identifying whether the abnormal image pattern corresponds to a pathological anomaly using the identification model.

## Claims

### Claim 1

A method performed by a computing system for detecting pathological anomalies in a digital pathology image, comprising:
providing a digital pathology image to the computing system; and
analyzing the digital pathology image using an identification module arranged on the computing system, wherein the identification module uses a machine learning module to execute:
recognizing an object containing an abnormal image pattern using an identification model loaded in said identification module; and
identifying whether the abnormal image pattern corresponds to a pathological anomaly using the identification model.

### Claim 2 (depends on 1)

The method according to claim 1, wherein identifying whether the abnormal image pattern corresponds to a pathological anomaly comprises:
classifying the abnormal image pattern using a classifier in the identification model to classify the abnormal image pattern in accordance with at least two classes; and
determining whether the abnormal image pattern corresponds to a pathological anomaly based on the classification.

### Claim 3 (depends on 1)

The method according to claim 1, further comprising providing the identification model using a machine learning algorithm trained on a plurality of annotated digital pathology images.

### Claim 4 (depends on 3)

The method according to claim 3, wherein providing the identification model comprises:
generating at least one first image of a tissue slide comprising stained morphological features of the pathological anomaly, wherein the tissue slide is stained with at least one biomarker and at least one first histochemical dye or at least one biomarker and at least one fluorescence dye or at least one fluorescent biomarker; and
annotating the at least one first image using the stained regions.

### Claim 5 (depends on 3)

The method according to claim 3, wherein providing the identification model comprises:
providing the annotated images to the machine learning module; and
training the identification model using different morphological features of the pathological anomalies to obtain the classifier.

### Claim 6 (depends on 3)

The method according to claim 3, wherein providing the identification model comprises:
providing a tissue slide with pathological anomalies;
selecting at least one biomarker and at least one first histochemical dye or at least one biomarker and at least one fluorescence dye or at least one fluorescent biomarker, specific for a respective pathological anomaly;
staining the tissue slide with the at least one biomarker and the at least one first histochemical dye or the at least one biomarker and the at least one fluorescence dye or the at least one fluorescent biomarker;
generating at least one first image of the tissue slide;
removing the first histochemical dye from the tissue slide;
staining the tissue slide with a second histochemical dye;
generating at least one second image of the tissue slide;
annotating the at least one second image of the tissue slide;
providing the annotated images to the machine learning module; and
training the identification model using different morphological features of the pathological anomalies to obtain the classifier.

### Claim 7 (depends on 5)

The method according to claim 5, wherein training the identification module comprises using clinical outcome.

### Claim 8 (depends on 6)

The method according to claim 6, wherein providing a tissue slide comprises selecting a tissue slide from a patient where patient data and/or clinical treatment and/or clinical outcome are known.

### Claim 9 (depends on 6)

The method according to claim 6, wherein the at least one fluorescence dye is a fluorescent direct stain.

### Claim 10 (depends on 6)

The method according to claim 6, wherein the biomarker is a morphological biomarker and/or a cancer specific biomarker and/or another disease, different from cancer, specific biomarker.

### Claim 11 (depends on 6)

The method according to claim 6, wherein removing the first histochemical dye from the tissue slide comprises using a detergent or a solvent or a bleach.

### Claim 12 (depends on 6)

The method according to claim 6, wherein the second histochemical dye are selected from a group comprising Haematoxylin, Eosin, van Gieson, Toluidine blue, Silver stain, Periodic acid-Schiff (PAS), Glycogen stain, Weigerts stain, Nissl stain, Golgi stain, Safranin, Oil Red, Prussian blue, Picro-Sirius Red, Mallary's trichome, Steiner Stain, Iron Hematoxylin and Fleugen stain.

### Claim 13 (depends on 1)

The method according to claim 1, further comprising providing a user interface for inputting a digital pathology image to the computing system and for outputting a display of detected pathological anomalies in the digital pathology image.

### Claim 14 (depends on 1)

The method according to claim 1, wherein the pathological anomaly is an infection or an inflammation or a cancer tumor.

### Claim 15 (depends on 2)

The method according to claim 2, wherein the at least one class is a cancer type and/or a cancer grade and/or a cancer stage.

### Claim 16 (depends on 1)

The method according to claim 1, wherein the machine learning module is a deep convolutional neural network.

### Claim 17 (depends on 3)

The method according to claim 3, wherein the machine learning algorithm is a deep convolutional neural network algorithm.

### Claim 18 (depends on 4)

The method according to claim 4, wherein the at least one first image of the tissue slide and the at least one second image of the tissue slide are images of a single tissue slide or a series of consecutive tissue slides.

### Claim 19 (depends on 6)

The method according to claim 6, wherein annotating the at least one second image of the tissue slide comprises layering the at least one first image of the tissue slide and the at least one second image of the tissue slide over another and using the stained region of the at least one first image of the tissue slide as mask for cropping the at least one second image of the tissue slide.

### Claim 20

A method for annotating a tissue slide image, comprising:
providing a tissue slide with pathological anomalies;
staining the tissue slide with at least one biomarker and at least one first histochemical dye or at least one biomarker and at least one fluorescence dye or at least one fluorescent biomarker, specific for a respective pathological anomaly;
generating at least one first image of the tissue slide;
removing the first histochemical dye from the tissue slide;
staining the tissue slide with a second histochemical dye;
generating at least one second image of the tissue slide; and
annotating the at least one second image of the tissue slide by using a stained region of the at least one first image of the tissue slide.

### Claim 21 (depends on 20)

The method according to claim 20, wherein providing a tissue slide comprises selecting a tissue slide from a patient where patient data and/or clinical treatment and/or clinical outcome are known.

### Claim 22 (depends on 20)

The method according to claim 20, wherein the at least one fluorescence dye is a fluorescent direct stain.

### Claim 23 (depends on 20)

The method according to claim 20, wherein the biomarker is a morphological biomarker and/or a cancer specific biomarker and/or another disease, different from cancer, specific biomarker.

### Claim 24 (depends on 20)

The method according to claim 20, wherein removing the first histochemical dye from the tissue slide comprises using a detergent or a solvent or a bleach.

### Claim 25 (depends on 20)

The method according to claim 20, wherein the second histochemical dye are selected from a group comprising Haematoxylin, Eosin, van Gieson, Toluidine blue, Silver stain, Periodic acid-Schiff (PAS), Glycogen stain, Weigerts stain, Nissl stain, Golgi stain, Safranin, Oil Red, Prussian blue, Picro-Sirius Red, Mallary's trichome, Steiner Stain, Iron Hematoxylin and Fleugen stain.

### Claim 26 (depends on 20)

The method according to claim 20, wherein the pathological anomaly is an infection or an inflammation or a cancer tumor.

### Claim 27 (depends on 20)

The method according to claim 20, wherein the at least one first image of the tissue slide and the at least one second image of the tissue slide are images of a single tissue slide or a series of consecutive tissue slides.

### Claim 28 (depends on 20)

The method according to claim 20, wherein annotating comprises layering the at least one first image of the tissue slide and the at least one second image of the tissue slide over another and using the stained region of the at least one first image of the tissue slide as mask for cropping the at least one second image of the tissue slide.

### Claim 29

A system for detecting pathological anomalies in a digital pathology image, comprising:
a digital pathology image receiving module; and
a digital pathology image analyzing module; wherein the digital pathology image analyzing module comprises an identification module, wherein the identification module comprises a machine learning module and an identification model,
wherein the machine learning module comprises:
a recognizing module for recognizing an object containing an abnormal image pattern using the identification model; and
an identifying module for identifying whether the abnormal image pattern corresponds to a pathological anomaly using the identification model.

### Claim 30 (depends on 29)

The system according to claim 29, wherein the identification model comprises
a classifier to classify the abnormal image pattern in accordance with at least two classes; and
a determining module for determining whether the abnormal image pattern corresponds to a pathological anomaly based on the classification.

### Claim 31 (depends on 29)

The system according to claim 29, further comprising an identification model providing module,
wherein the identification model providing module comprises a training module for training a machine learning algorithm based at a plurality of annotated digital pathology images.

### Claim 32 (depends on 29)

The system according to claim 29, further comprising:
a user interface comprising:
an input module for providing a digital pathology image to the system; and
an output module for displaying detected pathological anomalies in the digital pathology image.
