# US 2021/0174154 A1 · Interpretable deep machine learning for clinical radio;ogy

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174154 A1
- Document ID: US20210174154A1
- Status: Published application
- Kind: A1
- Application number: 17/265,862
- Filed: 2019-08-07
- Published: 2021-06-10
- Earliest claimed: 2018-08-07
- Assignees: Yale University
- Applicants: Yale University
- Inventors: James S. Duncan; Brian Letzen; Julius Chapiro; Clinton Wang
- Web page: https://patentgrep.com/patent/US20210174154A1

## CPC

- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06F 18/21: Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06F 18/24147: {Distances to closest patterns, e.g. nearest neighbour classification}
- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06F 18/2431: Multiple classes
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06T 7/0012: {Biomedical image inspection}
- G06T 11/00: Two-dimensional [2D] image generation
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G01N 2800/60: Complex ways of combining multiple protein biomarkers for diagnosis
- G06N 3/048: Activation functions
- G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G01N: INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES (measuring or testing apparatus or processes other than immunoassay, involving enzymes or microorganisms C12M, C12Q)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

One aspect of the invention provides a computer-implemented method of identifying one or more clinical factors associated with an artificial intelligence prediction. The computer implemented method includes: applying a previously trained deep neural network to one or more images for a subject to produce a prediction, the previously trained deep neural network comprising a plurality of nodes; comparing outputs of the nodes to previously identified patterns of node outputs for a plurality of individual clinical factors; and identifying which of the plurality of individual clinical factors are most correlated with the outputs of the nodes.

## Claims

### Claim 1

A computer-implemented method of identifying one or more clinical factors associated with an artificial intelligence prediction, the computer-implemented method comprising:
applying a previously trained deep neural network to one or more images for a subject to produce a prediction, the previously trained deep neural network comprising a plurality of nodes;
comparing outputs of the nodes to previously identified patterns of node outputs for a plurality of individual clinical factors; and
identifying which of the plurality of individual clinical factors are most correlated with the outputs of the nodes.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the previously identified patterns of node outputs for a plurality of individual clinical factors were generated by applying the previously trained deep neural network to one or more images having a single clinical factor.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating a contribution score for one or more of the plurality of individual clinical factors.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein the generating the contribution score step further comprises:
iteratively removing one or more of the multilayer nodes associated with the previously identified patterns of node outputs for one of the plurality of individual clinical factors from the previously trained deep neural network to produce an altered deep neural network;
applying the altered deep neural network to the one or more images to produce an altered prediction; and
identifying a difference between the altered prediction and the prediction.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein the generating the contribution score step further comprises:
quantifying the impact of removing the multilayer nodes associated with the previously identified patterns of node outputs for one of the plurality of individual clinical factors.

### Claim 6 (depends on 3)

The computer-implemented method of claim 3, wherein the generating the contribution score step further comprises:
quantifying the impact of removing one or more images having a single clinical factor on the prediction of the deep neural network.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising:
generating one or more altered images illustrating regions of the one or more images associated with the one or more individual clinical factors correlated with the outputs of the multilayer nodes.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, wherein the generating one or more altered images step further comprises:
identifying which of the multilayer node outputs are most correlated with each clinical factor;
comparing the multilayer node outputs for the one or more images used for prediction relative to the multilayer node outputs for the one or more images correlated with individual clinical factors;
designating each pixel or voxel in the one or more images used for prediction as significant to the one or more individual clinical factors if the multilayer node outputs associated with the pixel or voxel are correlated with the individual clinical factors and are comparable to the multilayer node outputs found in the one or more images correlated with individual clinical factors; and
applying an overlay for each of the one or more individual clinical factors corresponding to the pixels or voxels designated as significant to the individual clinical factor.

### Claim 9 (depends on 7)

The computer-implemented method of claim 7, wherein the generating one or more altered images step further comprises:
iteratively removing one or more pixels or voxels from the one or more images to generate one or more stripped images;
reapplying the previously trained deep neural network to the one or more stripped images;
identifying one or more differences in outputs of the multilayer nodes for the one or more stripped images relative to the outputs of the multilayer nodes for the one or more images;
comparing the one or more differences to the previously identified patterns of node outputs for the plurality of individual clinical factors;
designating a pixel or voxel as associated with the one or more individual clinical factors correlated with the outputs of the multilayer nodes if the differences are correlated with the previously identified patterns of node outputs for the plurality of individual clinical factors; and
applying an overlay corresponding to the pixels or voxels designated as associated with the one or more individual clinical factors.
