# US 2020/0093455 A1 · Method and system of building hospital-scale chest x-ray database for entity extraction and weakly-supervised classification and localization of common thorax diseases

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0093455 A1
- Document ID: US20200093455A1
- Status: Published application
- Kind: A1
- Application number: 16/495,012
- Filed: 2018-03-26
- Published: 2020-03-26
- Earliest claimed: 2017-03-24
- Assignees: The United States of America, as represented by the Secretary, Department of Health and; Human Services
- Applicants: The United States of America, as represented by the Secretary, Department of Health and Human Servic
- Inventors: Xiaosong Wang; Yifan Peng; Le Lu; Zhiyong Lu; Ronald M. Summers
- Web page: https://patentgrep.com/patent/US20200093455A1

## CPC

- G16H 30/40: for processing medical images, e.g. editing
- A61B 6/5217: {extracting a diagnostic or physiological parameter from medical diagnostic data}
- G06F 18/2155: {characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/048: Activation functions
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/0014: {using an image reference approach}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- A61B 5/08: Measuring devices for evaluating the respiratory organs
- A61B 5/7267: {involving training the classification device}
- G06T 2207/10116: X-ray image
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30061: Lung
- G06T 2210/12: Bounding box
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A new chest X-ray database, referred to as “ChestX-ray8”, is disclosed herein, which comprises over 100,000 frontal view X-ray images of over 32,000 unique patients with the text-mined eight disease image labels (where each image can have multi-labels), from the associated radiological reports using natural language processing. We demonstrate that these commonly occurring thoracic diseases can be detected and spatially-located via a unified weakly supervised multi-label image classification and disease localization framework, which is validated using our disclosed dataset.

## Claims

### Claim 1

A method comprising:
providing a chest x-ray image of a subject; and
by a computing system, analyzing the chest x-ray image of the subject by applying a unified weakly supervised multi-label image classification and disease localization framework to the chest x-ray image of the subject and determining the presence and anatomical locations of one or more thoracic diseases in the subject based on the analysis of the chest x-ray image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising generating an automated radiological report corresponding to the x-ray image of the subject based on the determined presence and anatomical locations of the one or more thoracic diseases in the subject.

### Claim 3 (depends on 1)

The method of claim 1, wherein the determining utilizes a database containing a plurality of chest x-ray images and a plurality of corresponding thoracic diseases and locations associated with the plurality of chest x-ray images, the plurality of corresponding thoracic diseases and locations having been text-mined from radiological reports corresponding to the plurality of chest x-ray images using natural language processing.

### Claim 4 (depends on 1)

The method of claim 1, wherein the one or more thoracic diseases comprise Atelectasis, Cardiomegaly, Effusion, Infiltration, Mass, Nodule, Pneumonia, and Pneumathorax.

### Claim 5 (depends on 1)

The method of claim 1, wherein the anatomical locations of the one or more thoracic diseases are identified with one or more bounding boxes localized relative to the chest x-ray images.

### Claim 6 (depends on 1)

The method of claim 1, wherein Deep Convolutional Neural Network (DCNN) architectures are used for the weakly-supervised disease localization.

### Claim 7 (depends on 1)

The method of claim 1, wherein the unified weakly supervised multi-label image classification and disease localization framework comprises a multi-label DCNN classification model, wherein the DCNN comprises a transition layer, a global pooling layer, a prediction layer and a multi-label classification loss layer in the end after a last convolutional layer.

### Claim 8 (depends on 7)

The method of claim 7, wherein spatial locations of diseases are determined using a combination of deep activations from the transition layer and weights of the prediction layer.

### Claim 9 (depends on 7)

The method of claim 7, wherein the transition layer transforms the activations from previous layers into a uniform dimension of output.

### Claim 10

A system comprising:
a computer processor; and
a data storage device that stores a database comprising:
a plurality of chest x-ray images; and
disease labels and locations associated with each chest x-ray image;
wherein the disease labels and locations are mined from radiological reports associated with the chest x-ray images, the disease labels identifying thoracic diseases indicated by the corresponding chest x-ray images, and the locations identifying anatomical areas in the chest x-ray images where the indicated thoracic diseases are located;
wherein the computer processor is operable to receive or generate a chest x-ray image of a subject and to determine presence and anatomical locations of one or more thoracic diseases in the subject based on automated analysis of the chest x-ray image of the subject using a unified weakly supervised multi-label image classification and disease localization framework.

### Claim 11 (depends on 10)

The system of claim 10, wherein the system is further operable to generate a radiological report corresponding to the x-ray image of the subject based on the determined presence and anatomical locations of the one or more thoracic diseases in the subject.

### Claim 12 (depends on 10)

The system of claim 10, wherein the thoracic diseases comprise Atelectasis, Cardiomegaly, Effusion, Infiltration, Mass, Nodule, Pneumonia, and Pneumathorax.

### Claim 13 (depends on 10)

The system of claim 10, wherein the anatomical locations of the one or more thoracic diseases are identified with one or more bounding boxes localized relative to the chest x-ray images.

### Claim 14 (depends on 10)

The system of claim 10, wherein the system uses Deep Convolutional Neural Network (DCNN) architectures for weakly-supervised object localization.

### Claim 15 (depends on 10)

The system of claim 10, wherein the unified weakly supervised multi-label image classification and disease localization framework comprises a multi-label DCNN classification model, wherein the DCNN comprises a transition layer, a global pooling layer, a prediction layer and a multi-label classification loss layer.

### Claim 16 (depends on 15)

The system of claim 15, wherein spatial locations of diseases are determined using a combination of deep activations from the transition layer and weights of the prediction layer.

### Claim 17 (depends on 10)

The system of claim 10, further comprising an x-ray imaging device that generates the chest x-ray of the subject.

### Claim 18 (depends on 1)

A computer readable storage device comprising computer-executable instructions for performing the method of claim 1.

### Claim 19 (depends on 10)

A computer readable storage device comprising the database of claim 10.

### Claim 20 (depends on 1)

The method of claim 1, further comprising treating the subject for the determined thoracic diseases.