# US 2020/0286614 A1 · A system and method for automated labeling and annotating unstructured medical datasets

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286614 A1
- Document ID: US20200286614A1
- Status: Published application
- Kind: A1
- Application number: 16/644,888
- Filed: 2018-09-10
- Published: 2020-09-10
- Earliest claimed: 2017-09-08
- Assignees: The General Hospital Corporation
- Applicants: The General Hospital Corporation
- Inventors: Synho Do
- Web page: https://patentgrep.com/patent/US20200286614A1

## CPC

- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06F 18/2431: Multiple classes
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06V 10/42: Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
- G06V 10/751: {Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/20: in augmented reality scenes
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G16H 20/40: relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
- G16H 30/40: for processing medical images, e.g. editing
- G06V 2201/031: of internal organs
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Supervised and unsupervised learning schemes may be used to automatically label medical images for use in deep learning applications. Large labeled datasets may be generated from a small initial training set using an iterative snowball sampling scheme. A machine learning powered automatic organ classifier for imaging datasets, such as CT datasets, with a deep convolutional neural network (CNN) followed by an organ dose calculation is also provided. This technique can be used for patient-specific organ dose estimation since the locations and sizes of organs for each patient can be calculated independently.

## Claims

### Claim 1

A method for automatically processing unstructured medical imaging data to generate classified images, the method comprising:
a) acquiring medical image data of a subject;
b) subjecting the medical image data of the subject to a neural network to generate classified image data;
c) comparing the classified image data to a confidence test;
d) upon determining that the classified image data does not pass the confidence test, subjecting the classified image data to a variational autoencoder (VAE) that implements a snowball sampling algorithm to refine the classified image data by representing features of the classified image data into latent space with a Gaussian distribution;
e) repeating steps c) and d) until the classified image data passes the confidence test; and
f) generating annotated images from the classified image data.

### Claim 2 (depends on 1)

The method of claim 1 wherein the Gaussian distribution is achieved using a Gaussian mixture models (GMMs) to perform binary clustering within each class across the classified image data.

### Claim 3 (depends on 1)

The method of claim 1 wherein the confidence test includes a threshold of at least one of speed, classification accuracy, reproducibility, or efficacy.

### Claim 4 (depends on 1)

The method of claim 1 wherein the classified image data includes at least one of body region labels, a body organ labels, an organ label, an image feature label, or a condition label.

### Claim 5 (depends on 1)

The method of claim 1 further comprising training the neural network by selecting an initial seed sample size to generate the training dataset.

### Claim 6

A method for automatic labeling and annotation for unstructured medical datasets with snowball sampling comprising:
a) acquiring images of a region of a subject and labeling the images to generate a training dataset with the images;
b) training a convolutional neural network with the training dataset;
c) classifying unlabeled images using the trained network;
d) determining if a performance threshold is exceeded for the classified images; and
e) refining the dataset if the threshold is not exceeded by using a variational autoencoder to label the unlabeled images to create labeled images and updating the dataset with the labeled images.

### Claim 7 (depends on 6)

The method of claim 6 wherein the variational autoencoder generates latent vectors approximating a unit Gaussian distribution in order to minimize generative losses.

### Claim 8 (depends on 6)

The method of claim 6 wherein the variational autoencoder includes an encoder with a multilayer perceptron neural network allowing it to map an input to a latent representation, and a decoder that maps the latent representation to a reconstructed input value.

### Claim 9 (depends on 6)

The method of claim 6 further comprising selecting an initial seed sample size to generate the training dataset.

### Claim 10 (depends on 6)

The method of claim 6 wherein the performance threshold includes a threshold of at least one of speed, classification accuracy, reproducibility, or efficacy.

### Claim 11 (depends on 6)

The method of claim 6 wherein classifying unlabeled images includes at least one of identifying a body region, a body organ, an organ, an image feature, or a condition.

### Claim 12 (depends on 6)

The method of claim 6 wherein snowball sampling includes refining the dataset at least twice.

### Claim 13

A system for automatic labeling and annotation for unstructured medical datasets from medical images with snowball sampling, the system comprising:
a) a computer system configured to:
i) acquire images of a region of a subject and label the images to generate a training dataset with the images;
ii) train a convolutional neural network with the training dataset;
iii) classify unlabeled images using the trained network;
iv) determine if a performance threshold is exceeded for the classified images; and
v) refine the dataset if the threshold is not exceeded by using a variational autoencoder to label the unlabeled images to create labeled images and updating the dataset with the labeled images.

### Claim 14 (depends on 13)

The system of claim 13 wherein the variational autoencoder generates latent vectors approximating a unit Gaussian distribution in order to minimize generative losses.

### Claim 15 (depends on 13)

The system of claim 13 wherein the variational autoencoder includes an encoder with a multilayer perceptron neural network allowing it to map an input to a latent representation, and a decoder that maps the latent representation to a reconstructed input value.

### Claim 16 (depends on 13)

The system of claim 13 further comprising selecting an initial seed sample size to generate the training dataset.

### Claim 17 (depends on 13)

The system of claim 13 wherein the performance threshold includes a threshold of at least one of speed, classification accuracy, reproducibility, or efficacy.

### Claim 18 (depends on 13)

The system of claim 13 wherein classifying unlabeled images includes at least one of identifying a body region, a body organ, an organ, an image feature, or a condition.

### Claim 19 (depends on 13)

The system of claim 13 wherein snowball sampling includes refining the dataset at least twice.
