# US 10,499,857 B1 · Medical protocol change in real-time imaging

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,499,857 B1
- Document ID: US10499857B1
- Status: Granted
- Kind: B1
- Application number: 15/709,293
- Filed: 2017-09-19
- Granted: 2019-12-10
- Earliest claimed: 2017-09-19
- Assignees: Deepradiology Inc.
- Applicants: Deepradiology, Inc.
- Inventors: Kim Nguyen; Robert Burnham Lufkin; Jameson Tyler Merkow
- Web page: https://patentgrep.com/patent/US10499857B1

## CPC

- A61B 5/7267: {involving training the classification device}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06T 7/0012: {Biomedical image inspection}
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30004: Biomedical image processing
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA

## Abstract

Methods and systems for medical diagnosis by machine learning are disclosed. Imaging data obtained from different medical techniques can be used as a training set for a machine learning method, to allow diagnosis of medical conditions in a faster a more efficient manner. A three-dimensional convolutional neural network can be employed to interpret volumetric data available from multiple scans of a patient. The imaging data can be analyzed in real-time to prescribe additional testing while the patient is still within the medical facility.

## Claims

### Claim 1

A device comprising:
a processor and memory configured to implement the steps of:
receiving imaging data relating to a body part of a human patient, taken at a medical imaging facility;
classifying by machine learning the imaging data into a normal or an abnormal class;
if the imaging data is classified as abnormal, suggesting collection of additional imaging data of the body part while the human patient is still present at the medical imaging facility;
for the abnormal class, spatially localizing an abnormality within the imaging data of the body part;
categorizing by machine learning the abnormality within a category; and
implement computer code of a convolutional neural network (CNN), the CNN comprising a plurality of layers in a sequence, each layer comprising a plurality of nodes, wherein:
each node of each layer is connected to at least one other node of a subsequent or preceding layer in the sequence,
each node accepts an input value and outputs an output value,
the classifying, spatially localizing and categorizing are by the CNN,
the imaging data comprises a training set of correctly diagnosed imaging data, the nodes in the CNN comprise parameters, and the CNN is trained on the training set to optimize the node parameters,
the plurality of layers comprises convolutional layers, rectified linear unit layers, non-linear pooling layers and fully connected layers, and
the non-linear pooling layers are max pooling layers that partition at least one image of the imaging data into non-overlapping sub-regions and output a maximum value for each sub-region.

### Claim 2 (depends on 1)

The device of claim 1, wherein the CNN is trained by deep supervision based on known anatomy of the body part.

### Claim 3 (depends on 2)

The device of claim 2, wherein:
the imaging data is three-dimensional and comprises anatomical annotations, and
the CNN is a three-dimensional CNN (3DCNN) trained to recognize three-dimensionality of the imaging data, and exploit the anatomical annotations.

### Claim 4 (depends on 3)

The device of claim 3, wherein the imaging data comprises a set of images, the classifying comprises assigning a normal or abnormal label to at least one image of the set, and the label is propagated by the computer to at least one other image of the set previously without label.

### Claim 5 (depends on 1)

The device of claim 1, wherein the imaging data comprises X-ray slides, computerized tomography, magnetic-resonance, diffusion-tensor (DT), functional MR, gene-expression data, dermatological images, or optical imaging of tissue slices.

### Claim 6 (depends on 1)

The device of claim 1, wherein prescribing collection of additional imaging data comprises specifying a region of the body part and limiting the additional imaging data to the specified region.

### Claim 7 (depends on 1)

The device of claim 1, wherein prescribing collection of additional imaging data comprises prescribing a different imaging technique or prescribing collection at an increased resolution.

### Claim 8 (depends on 1)

The device of claim 1, wherein the processor and memory are further configured to repeat the steps of: receiving imaging data, classifying by machine learning the imaging data, and suggesting collection of additional imaging data.

### Claim 9

A method comprising:
receiving, by a computer, imaging data relating to a body part of a human patient;
classifying by machine learning the imaging data into a normal or an abnormal class;
if the imaging data is classified as abnormal, suggesting collection of additional imaging data of the body part while the human patient is still present at the medical imaging facility;
implementing in the computer a convolutional neural network (CNN), the CNN comprising a plurality of layers in a sequence, each layer comprising a plurality of nodes, wherein:
each node of each layer is connected to at least one other node of a subsequent or preceding layer in the sequence,
each node accepts an input value and outputs an output value,
the classifying, spatially localizing and categorizing are by the CNN,
the plurality of layers comprises convolutional layers, rectified linear unit layers, non-linear down-sampling layers and fully connected layers, and
the non-linear down-sampling layers are max pooling layers;
partitioning, by the max pooling layers, at least one image of the imaging data into non-overlapping sub-regions; and
outputting a maximum value for each sub-region.

### Claim 10 (depends on 9)

The method of claim 9, further comprising for the abnormal class, spatially localizing by machine learning an abnormality within the imaging data of the body part.

### Claim 11 (depends on 10)

The method of claim 10, further comprising categorizing by machine learning the abnormality within a category.

### Claim 12 (depends on 9)

The method of claim 9, wherein the imaging data comprises a training set of correctly diagnosed imaging data, the nodes in the CNN comprise parameters, and further comprising optimizing the node parameters by training the CNN on the training set.

### Claim 13 (depends on 9)

The method of claim 9, further comprising training the CNN by deep supervision based on known anatomy of the body part.

### Claim 14 (depends on 13)

The method of claim 13, wherein the imaging data is three-dimensional and comprises anatomical annotations, the CNN is a three-dimensional CNN (3DCNN), and further comprising training the 3DCNN to recognize three-dimensionality of the imaging data and exploit the anatomical annotations.

### Claim 15 (depends on 9)

The method of claim 9, wherein the imaging data comprises a batch of images, the classifying comprises assigning a normal or abnormal label to at least one image of the batch, and further comprising propagating the label to at least one other image of the batch previously without label.

### Claim 16 (depends on 9)

The method of claim 9, wherein the imaging data comprises X-ray slides, computerized tomography, magnetic-resonance, diffusion-tensor (DT), functional MR, gene-expression data, dermatological images, or optical imaging of tissue slices.

### Claim 17 (depends on 9)

The method of claim 9, wherein prescribing collection of additional imaging data comprises specifying a region of the body part and limiting the additional imaging data to the specified region.

### Claim 18 (depends on 9)

The method of claim 9, wherein prescribing collection of additional imaging data comprises prescribing a different imaging technique or prescribing collection at an increased resolution.

### Claim 19 (depends on 9)

The method of claim 9, further comprising repeating the steps of: receiving imaging data, classifying by machine learning the imaging data, and suggesting collection of additional imaging data.

### Claim 20 (depends on 15)

The method of claim 15, further comprising:
processing by the CNN each image of the batch of images according to a tensor having at least a batch dimension, and at least two spatial dimensions; and
reshaping the tensor to a variable batch dimension.