# US 10,482,313 B2 · Method and system for classification of endoscopic images using deep decision networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,482,313 B2
- Document ID: US10482313B2
- Status: Granted
- Kind: B2
- Application number: 15/756,263
- Filed: 2016-09-29
- Published: 2018-08-30
- Granted: 2019-11-19
- Earliest claimed: 2015-09-30
- Assignees: Siemens Healthcare GmbH
- Applicants: Siemens Healthcare GmbH; Siemens Corporation
- Inventors: Venkatesh N. Murthy; Vivek Kumar Singh; Shanhui Sun; Subhabrata Bhattacharya; Kai Ma; Ali Kamen; Bogdan Georgescu; Terrence Chen; Dorin Comaniciu
- Web page: https://patentgrep.com/patent/US10482313B2

## CPC

- G06T 7/0012: {Biomedical image inspection}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 20/698: {Matching; Classification}
- G06T 2207/10068: Endoscopic image
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

A method and system for classification of endoscopic images is disclosed. An initial trained deep network classifier is used to classify endoscopic images and determine confidence scores for the endoscopic images. The confidence score for each endoscopic image classified by the initial trained deep network classifier is compared to a learned confidence threshold. For endoscopic images with confidence scores higher than the learned threshold value, the classification result from the initial trained deep network classifier is output. Endoscopic images with confidence scores lower than the learned confidence threshold are classified using a first specialized network classifier built on a feature space of the initial trained deep network classifier.

## Claims

### Claim 1

A method for classifying endoscopic images, comprising:
classifying an endoscopic image and determining a confidence score for the endoscopic image using an initial trained deep network classifier;
comparing the confidence score for the endoscopic image to a learned confidence threshold;
in response to a determination that the confidence score for the endoscopic image is higher than the learned confidence threshold, outputting the classification of the endoscopic image by the initial trained deep network classifier; and
in response to a determination that the confidence score for the endoscopic image is not higher than the learned confidence threshold, classifying the endoscopic image with a first specialized network classifier built on a feature space of the initial trained deep network classifier,
wherein the learned confidence threshold is a confidence score for which no training samples in a validation dataset of training data were incorrectly classified by the initial trained deep network classifier.

### Claim 2 (depends on 1)

The method of claim 1, wherein comparing the confidence score for the endoscopic image to a learned confidence threshold comprises:
comparing the confidence score for the endoscopic image to a first leaned confidence threshold if the endoscopic image is classified into a first class by the initial trained deep network classifier; and
comparing the confidence score for the endoscopic image to a second learned confidence threshold if the endoscopic image is classified into a second class by the initial trained deep network classifier.

### Claim 3 (depends on 1)

The method of claim 1, wherein an input layer of the initial trained deep network classifier inputs raw pixel data of the endoscopic image, and an input layer of the first specialized network classifier inputs a feature vector output by a hidden layer of the initial trained deep network classifier.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first specialized network classifier is trained based on only training samples classified by the initial trained deep network classifier having confidence scores lower than the learned confidence threshold.

### Claim 5 (depends on 1)

The method of claim 1, further comprising, in response to classifying the endoscopic image with the first specialized network classifier:
comparing a second confidence value determined for the endoscopic image by the first specialized network classifier to a second learned confidence threshold;
in response to a determination that the second confidence score for the endoscopic image is higher than the second learned confidence threshold, outputting the classification of the endoscopic image by the first specialized network classifier; and
in response to a determination that the second confidence score for the endoscopic image is not higher than the second learned confidence threshold, classifying the endoscopic image with a second specialized network classifier built on a feature space of the first specialized network classifier.

### Claim 6 (depends on 5)

The method of claim 5, wherein the second learned confidence threshold is a confidence score for which no training samples in a validation dataset of training data were incorrectly classified by the first specialized network classifier.

### Claim 7 (depends on 5)

The method of claim 5, wherein an input layer of the second specialized network classifier inputs a feature vector output by a hidden layer of the first specialized network classifier.

### Claim 8 (depends on 7)

The method of claim 7, wherein the second specialized network classifier is trained based on only training samples classified by the first specialized network classifier having second confidence scores lower than the second learned confidence threshold.

### Claim 9 (depends on 1)

The method of claim 1, wherein the endoscopic image is a frame of a colonoscopy video, and classifying the endoscopic image comprises:
classifying the endoscopic image as positive or negative, wherein a positive classification indicates that a polyp is present in the endoscopic image and a negative classification indicates the no polyp is present in the endoscopic image.

### Claim 10 (depends on 1)

The method of claim 1, wherein the endoscopic image is a confocal laser endomicroscopy image of brain tumor tissue, and classifying the endoscopic image comprises:
classifying the endoscopic image as glioblastoma tissue or meningioma tissue.

### Claim 11 (depends on 1)

The method of claim 1, wherein classifying an endoscopic image and determining a confidence score for the endoscopic image using an initial trained deep network classifier comprises classifying the endoscopic image as one of a plurality of classes comprising three or more classes, and classifying the endoscopic image with a first specialized network classifier built on a feature space of the initial trained deep network classifier comprises:
identifying a cluster for the endoscopic image from a plurality of learned clusters, each cluster corresponding to a respective subset of the plurality of classes;
selecting the first specialized network classifier based on the identified cluster from a plurality of specialized network classifiers, each trained for a respective one of the plurality of learned clusters; and
classifying the endoscopic image as one of the subset of classes corresponding to the identified cluster with the first specialized network classifier.

### Claim 12 (depends on 11)

The method of claim 11, wherein the initial trained deep network classifier is fine-tuned using a weighted contrastive loss function to penalize miss-classifications of training images into classes not within the same cluster as ground truth classes for the training images with a greater penalty as compared with miss-classifications of training images into classes within the same cluster as ground truth classes for the training images.

### Claim 13

A method of classifying endoscopic images, comprising:
receiving a plurality of endoscopic images;
classifying each of the plurality of endoscopic images and determining a confidence score for each of the plurality of endoscopic images using an initial trained deep network classifier;
comparing the confidence score for each of the plurality of endoscopic images to a learned confidence threshold to determine a highly confident subset of the plurality of endoscopic images and a confusion subset of the plurality of endoscopic images;
outputting classification results from the initial trained deep network classifier for the highly confident subset of the plurality of endoscopic images; and
classifying each of the confusion subset of the plurality of endoscopic images using one or more specialized network classifiers,
wherein classifying each of the confusion subset of the plurality of endoscopic images using one or more specialized network classifiers comprises:
classifying of the endoscopic images in the confusion subset using a first specialized network classifier built on a feature space of the initial trained deep network classifier.

### Claim 14 (depends on 13)

The method of claim 13, wherein classifying each of the confusion subset of the plurality of endoscopic images using one or more specialized network classifiers further comprises:
comparing a second confidence score for each of the endoscopic images in the confusion subset determined by the first specialized network classifier to a second learned confidence threshold to determine highly confident cases and confusion cases of the endoscopic images classified by the first specialized network classifier;
outputting classification results from the first specialized network classifier for the highly confident cases of the endoscopic images classified by the first specialized network classifier; and
classifying the confusion cases of the endoscopic images classified by the first specialized network classifier by one or more subsequent specialized network classifiers, wherein each subsequent specialized network classifier is built on a feature space of the previous specialized network classifier and each subsequent specialized network classifier classifies only confusion cases of endoscopic images classified by the previous specialized network classifier.

### Claim 15

An apparatus for classifying endoscopic images, comprising:
means for classifying endoscopic images and determining confidence scores for the endoscopic images using an initial trained deep network classifier;
means for comparing the confidence score for each endoscopic image classified by the initial trained deep network classifier to a learned confidence threshold; and
means for classifying endoscopic images with confidence scores lower than the learned confidence threshold with a first specialized network classifier built on a feature space of the initial trained deep network classifier
wherein an input layer of the initial trained deep network classifier inputs raw pixel data of endoscopic images, and an input layer of the first specialized network classifier inputs a feature vector output by a hidden layer of the initial trained deep network classifier.

### Claim 16 (depends on 10)

The apparatus of claim 10, wherein the means for comparing the confidence score for each endoscopic image classified by the initial trained deep network classifier to a learned confidence threshold comprises:
comparing the confidence score for each endoscopic image classified into a first class by the initial trained deep network classifier to a first leaned confidence threshold; and
comparing the confidence score for each endoscopic image classified into a second class by the initial trained deep network classifier to a second leaned confidence threshold.

### Claim 17 (depends on 10)

The apparatus of claim 10, wherein the learned confidence threshold is a confidence score for which no training samples in a validation dataset of training data were incorrectly classified by the initial trained deep network classifier.

### Claim 18 (depends on 10)

The apparatus of claim 10, wherein the first specialized network classifier is trained based on only training samples classified by the initial trained deep network classifier having confidence scores lower than the learned confidence threshold.

### Claim 19 (depends on 10)

The apparatus of claim 10, further comprising:
means for comparing, for each of the endoscopic images classified by the first specialized network classifier, a second confidence value determined by the first specialized network classifier to a second learned confidence threshold; and
means for classifying endoscopic images with second confidence values lower than the second learned confidence threshold using a second specialized network classifier built on a feature space of the first specialized network classifier.

### Claim 20 (depends on 19)

The apparatus of claim 19, wherein the second learned confidence threshold is a confidence score for which no training samples in a validation dataset of training data were incorrectly classified by the first specialized network classifier.

### Claim 21 (depends on 20)

The apparatus of claim 20, wherein the second specialized network classifier is trained based on only training samples classified by the first specialized network classifier having second confidence scores lower than the second learned confidence threshold.

### Claim 22 (depends on 19)

The apparatus of claim 19, wherein an input layer of the second specialized network classifier inputs a feature vector output by a hidden layer of the first specialized network classifier.

### Claim 23

A non-transitory computer readable medium storing computer program instructions for classifying endoscopic images, the computer program instructions when executed by a processor perform operations comprising:
classifying an endoscopic image and determining a confidence score for the endoscopic image using an initial trained deep network classifier;
comparing the confidence score for the endoscopic image to a learned confidence threshold;
in response to a determination that the confidence score for the endoscopic image is higher than the learned confidence threshold, outputting the classification of the endoscopic image by the initial trained deep network classifier; and
in response to a determination that the confidence score for the endoscopic image is not higher than the learned confidence threshold, classifying the endoscopic image with a first specialized network classifier built on a feature space of the initial trained deep network classifier
wherein an input layer of the second specialized network classifier inputs a feature vector output by a hidden layer of the first specialized network classifier.

### Claim 24 (depends on 23)

The non-transitory computer readable medium of claim 23, wherein comparing the confidence score for the endoscopic image to a learned confidence threshold comprises:
comparing the confidence score for the endoscopic image to a first leaned confidence threshold if the endoscopic image is classified into a first class by the initial trained deep network classifier; and
comparing the confidence score for the endoscopic image to a second learned confidence threshold if the endoscopic image is classified into a second class by the initial trained deep network classifier.

### Claim 25 (depends on 23)

The non-transitory computer readable medium of claim 23, wherein the learned confidence threshold is a confidence score for which no training samples in a validation dataset of training data were incorrectly classified by the initial trained deep network classifier.

### Claim 26 (depends on 23)

The non-transitory computer readable medium of claim 23, wherein an input layer of the initial trained deep network classifier inputs raw pixel data of the endoscopic image, and an input layer of the first specialized network classifier inputs a feature vector output by a hidden layer of the initial trained deep network classifier.

### Claim 27 (depends on 23)

The non-transitory computer readable medium of claim 23, wherein the first specialized network classifier is trained based on only training samples classified by the initial trained deep network classifier having confidence scores lower than the learned confidence threshold.

### Claim 28 (depends on 23)

The non-transitory computer readable medium of claim 23, further comprising, in response to classifying the endoscopic image with the first specialized network classifier:
comparing a second confidence value determined for the endoscopic image by the first specialized network classifier to a second learned confidence threshold;
in response to a determination that the second confidence score for the endoscopic image is higher than the second learned confidence threshold, outputting the classification of the endoscopic image by the first specialized network classifier; and
in response to a determination that the second confidence score for the endoscopic image is not higher than the second learned confidence threshold, classifying the endoscopic image with a second specialized network classifier built on a feature space of the first specialized network classifier.

### Claim 29 (depends on 28)

The non-transitory computer readable medium of claim 28, wherein the second learned confidence threshold is a confidence score for which no training samples in a validation dataset of training data were incorrectly classified by the first specialized network classifier.

### Claim 30 (depends on 28)

The non-transitory computer readable medium of claim 28, wherein the second specialized network classifier is trained based on only training samples classified by the first specialized network classifier having second confidence scores lower than the second learned confidence threshold.

### Claim 31 (depends on 23)

The non-transitory computer readable medium of claim 23, wherein the endoscopic image is a frame of a colonoscopy video, and classifying the endoscopic image comprises:
classifying the endoscopic image as positive or negative, wherein a positive classification indicates that a polyp is present in the endoscopic image and a negative classification indicates the no polyp is present in the endoscopic image.

### Claim 32 (depends on 23)

The non-transitory computer readable medium of claim 23, wherein the endoscopic image is a confocal laser endomicroscopy image of brain tumor tissue, and classifying the endoscopic image comprises:
classifying the endoscopic image as glioblastoma tissue or meningioma tissue.