# US 2022/0068444 A1 · Multi-model medical scan analysis system and methods for use therewith

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0068444 A1
- Document ID: US20220068444A1
- Status: Published application
- Kind: A1
- Application number: 17/393,726
- Filed: 2021-08-04
- Published: 2022-03-03
- Earliest claimed: 2018-11-21
- Assignees: Enlitic, Inc.
- Applicants: Enlitic, Inc.
- Inventors: Jordan Prosky; Li Yao; Eric C. Poblenz; Kevin Lyman; Ben Covington; Anthony Upton
- Web page: https://patentgrep.com/patent/US20220068444A1

## CPC

- G06Q 10/06315: {Needs-based resource requirements planning or analysis}
- A61B 5/7264: {Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems}
- A61B 6/5205: {involving processing of raw data to produce diagnostic data}
- A61B 6/563: {involving image data transmission via a network}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/0484: for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
- G06F 9/542: {Event management; Broadcasting; Multicasting; Notifications}
- G06F 16/245: Query processing
- G06F 18/2115: by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06F 18/2415: based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
- G06F 18/24155: {Bayesian classification}
- G06F 18/41: {Interactive pattern learning with a human teacher}
- G06F 21/6254: {by anonymising data, e.g. decorrelating personal data from the owner's identification}
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 5/04: Inference or reasoning models
- G06N 5/045: Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence
- G06N 20/00: Machine learning
- G06N 20/20: Ensemble learning
- G06Q 20/14: specially adapted for billing systems
- G06Q 20/145: {Payments according to the detected use or quantity}
- G06T 3/40: Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- G06T 5/94: based on local image properties, e.g. for local contrast enhancement
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/0014: {using an image reference approach}
- G06T 7/10: Segmentation; Edge detection (motion-based segmentation G06T7/215)
- G06T 7/11: Region-based segmentation
- G06T 7/187: involving region growing; involving region merging; involving connected component labelling
- G06T 7/44: using image operators, e.g. filters, edge density metrics or local histograms
- G06T 7/97: {Determining parameters from multiple pictures (depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85)}
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06T 11/26: Drawing of charts or graphs
- G06T 12/20: Inverse problem, i.e. transformations from projection space into object space
- G06V 10/225: {based on a marking or identifier characterising the area}
- G06V 10/25: Determination of region of interest [ROI] or a volume of interest [VOI]
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 30/19173: {Classification techniques}
- G06V 40/171: {Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships}
- G16H 10/20: for electronic clinical trials or questionnaires
- G16H 10/60: for patient-specific data, e.g. for electronic patient records
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 40/20: for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/50: for simulation or modelling of medical disorders
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- A61B 5/7267: {involving training the classification device}
- A61B 6/032: {Transmission computed tomography [CT]}
- A61B 6/5211: {involving processing of medical diagnostic data}
- A61B 6/5217: {extracting a diagnostic or physiological parameter from medical diagnostic data}
- A61B 6/5258: {involving detection or reduction of artifacts or noise}
- A61B 8/4416: {related to combined acquisition of different diagnostic modalities, e.g. combination of ultrasound and X-ray acquisitions}
- A61B 8/5207: {involving processing of raw data to produce diagnostic data, e.g. for generating an image}
- A61B 8/5215: {involving processing of medical diagnostic data}
- A61B 8/5269: {involving detection or reduction of artifacts}
- G06F 18/2111: by using evolutionary computational techniques, e.g. genetic algorithms
- G06F 18/24: Classification techniques
- G06F 40/295: Named entity recognition
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06Q 50/22: Social work or social welfare, e.g. community support activities or counselling services
- G06T 7/70: Determining position or orientation of objects or cameras (camera calibration G06T7/80)
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/10048: Infrared image
- G06T 2207/10072: Tomographic images
- G06T 2207/10081: Computed x-ray tomography [CT]
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/10116: X-ray image
- G06T 2207/10132: Ultrasound image
- G06T 2207/20076: Probabilistic image processing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30004: Biomedical image processing
- G06T 2207/30008: Bone
- G06T 2207/30016: Brain
- G06T 2207/30061: Lung
- G06V 30/194: References adjustable by an adaptive method, e.g. learning
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H 50/30: for calculating health indices; for individual health risk assessment
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- Y02A 90/10: Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
- G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- A61B: DIAGNOSIS; SURGERY; IDENTIFICATION (usefulness limited to only animals A61D)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- Y02A: TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE

## Abstract

A multi-model medical scan analysis system is operable to generate a generic model by performing a training step on image data of a plurality of medical scans and corresponding labeling data. A plurality of fine-tuned models are generated by performing a fine-tuning step on the generic model. Abnormality detection data is generated for a new medical scan by utilizing the generic model. A first one of the plurality of abnormality types that is detected in the new medical scan is determined based on a corresponding one of the plurality of probability values. Additional abnormality data is generated by performing a fine-tuned inference function on the image data of the new medical scan that utilizes one of the plurality of fine-tuned models that corresponds to the first one of the plurality of abnormality types. The additional abnormality data is transmitted for display.

## Claims

### Claim 1

A multi-model medical scan analysis system, comprising:
at least one processor; and
a memory that stores operational instructions that, when executed by the at least one processor, cause the multi-model medical scan analysis system to:
generate a generic model by performing a training step on image data of a plurality of medical scans and corresponding labeling data, wherein the corresponding labeling data of each of the plurality of medical scans includes indicates whether each of a plurality of abnormality types are present in the each of the plurality of medical scans;
generate a plurality of fine-tuned models by performing a fine-tuning step on the generic model, wherein each of the plurality of fine-tuned models corresponds to one of the plurality of abnormality types;
generate abnormality detection data for a new medical scan by utilizing the generic model, wherein the abnormality detection data indicates a plurality of probability values that each indicate a probability that a corresponding one of the plurality of abnormality types is present in the new medical scan;
determine a first one of the plurality of abnormality types that is detected in the new medical scan based on a corresponding one of the plurality of probability values;
generate additional abnormality data for the new medical scan by performing a fine-tuned inference function on the image data of the new medical scan, wherein the fine-tuned inference function utilizes one of the plurality of fine-tuned models that corresponds to the first one of the plurality of abnormality types; and
transmit the additional abnormality data for display.

### Claim 2 (depends on 1)

The multi-model medical scan analysis system of claim 1, wherein the additional abnormality data includes a new probability value indicating whether the first one of the plurality of abnormality types is present.

### Claim 3 (depends on 2)

The multi-model medical scan analysis system of claim 2, wherein the additional abnormality data indicates an increased confidence that first one of the plurality of abnormality types is present in the new medical scan in response to the new probability value indicating a higher probability value than the corresponding one of the plurality of probability values of the abnormality detection data.

### Claim 4 (depends on 2)

The multi-model medical scan analysis system of claim 2, wherein the additional abnormality data indicates that the first one of the plurality of abnormality types is not present in the new medical scan in response to the new probability value comparing unfavorably to a detection threshold.

### Claim 5 (depends on 1)

The multi-model medical scan analysis system of claim 1, wherein performing the fine-tuning step includes modifying a plurality of pre-tuned weights of the generic model.

### Claim 6 (depends on 5)

The multi-model medical scan analysis system of claim 5, wherein performing the fine-tuning step includes performing a training step on a subset of the plurality of medical scans that have corresponding labeling data indicating the first one of the plurality of abnormality types is present in the each of the subset of the plurality of medical scans, and wherein performing the training step includes utilizing a set of initial weights corresponding to the plurality of pre-tuned weights of the generic model.

### Claim 7 (depends on 1)

The multi-model medical scan analysis system of claim 1, wherein performing the fine-tuning step to generate one of the plurality of fine-tuned models corresponding to the first one of the plurality of abnormality types includes utilizing additional labeling data of a subset of the plurality of medical scans, wherein each of the subset of the plurality of medical scans have corresponding labeling data indicating the first one of the plurality of abnormality types is present in the each of the subset of the plurality of medical scans, wherein the additional labeling data indicates characterization labels characterizing the first one of the plurality of abnormality types in the each of the subset of the plurality of medical scans, and wherein the additional abnormality data indicates output data for the characterization labels.

### Claim 8 (depends on 7)

The multi-model medical scan analysis system of claim 7, wherein the output data indicates a plurality of probability values indicating whether each of a plurality of characterization types characterize the first one of the plurality of abnormality types.

### Claim 9 (depends on 8)

The multi-model medical scan analysis system of claim 8, wherein the output data indicates a subset of the plurality of characterization types determined to characterize the first one of the plurality of abnormality types in response to corresponding ones of the plurality of probability values comparing favorably to a characterization threshold.

### Claim 10 (depends on 1)

The multi-model medical scan analysis system of claim 1, wherein the operational instructions, when executed by the at least one processor, further cause the multi-model medical scan analysis system to:
determine a second one of the plurality of abnormality types by determining a second corresponding one of the plurality of probability values compares favorably to a detection threshold;
select a second one of the plurality of fine-tuned models that corresponds to the second one of the plurality of abnormality types;
generate second additional abnormality data for the new medical scan by performing a second fine-tuned inference function on the image data of the new medical scan, wherein the second fine-tuned inference function utilizes the second one of the plurality of fine-tuned models; and
transmit the second additional abnormality data to a client device for display via a display device in conjunction with the additional abnormality data.

### Claim 11 (depends on 1)

The multi-model medical scan analysis system of claim 1, wherein the abnormality detection data indicates region of interest data indicating a location of the first one of the plurality of abnormality types in the image data of the new medical scan, and wherein the operational instructions, when executed by the at least one processor, further cause the multi-model medical scan analysis system to:
select a sub-region of the image data of the new medical scan based on the region of interest data, wherein the fine-tuned inference function is performed only on the sub-region of the image data of the new medical scan.

### Claim 12 (depends on 11)

The multi-model medical scan analysis system of claim 11, wherein performing the fine-tuning step to generate the one of the plurality of fine-tuned models includes:
determining a subset of the plurality of medical scans that have corresponding labeling data indicating the first one of the plurality of abnormality types is present in the each of the subset of the plurality of medical scans;
selecting a plurality of sub-regions by selecting a sub-region of the image data of each of the subset of the plurality of medical scans based on region interest data of the corresponding labeling data indicating a location of the first one of the plurality of abnormality types in the image data of the each of the subset of the plurality of medical scans; and
performing a training step on the plurality of sub-regions to generate the one of the plurality of fine-tuned models.

### Claim 13 (depends on 1)

The multi-model medical scan analysis system of claim 1, wherein the plurality of medical scans correspond to chest x-rays.

### Claim 14

A method for execution by a multi-model medical scan analysis system, the method comprising:
generating a generic model by performing a training step on image data of a plurality of medical scans and corresponding labeling data, wherein the corresponding labeling data of each of the plurality of medical scans includes indicates whether each of a plurality of abnormality types are present in the each of the plurality of medical scans;
generating a plurality of fine-tuned models by performing a fine-tuning step on the generic model, wherein each of the plurality of fine-tuned models corresponds to one of the plurality of abnormality types;
generating abnormality detection data for a new medical scan by utilizing the generic model, wherein the abnormality detection data indicates a plurality of probability values that each indicate a probability that a corresponding one of the plurality of abnormality types is present in the new medical scan;
determining a first one of the plurality of abnormality types that is detected in the new medical scan based on a corresponding one of the plurality of probability values;
generating additional abnormality data for the new medical scan by performing a fine-tuned inference function on the image data of the new medical scan, wherein the fine-tuned inference function utilizes the one of the plurality of fine-tuned models that corresponds to the first one of the plurality of abnormality types; and
transmitting the additional abnormality data for display.

### Claim 15 (depends on 14)

The method of claim 14, wherein the additional abnormality data includes a new probability value indicating whether the first one of the plurality of abnormality types is present.

### Claim 16 (depends on 14)

The method of claim 14, wherein performing the fine-tuning step includes modifying a plurality of pre-tuned weights of the generic model.

### Claim 17 (depends on 14)

The method claim 14, wherein performing the fine-tuning step to generate one of the plurality of fine-tuned models corresponding to the first one of the plurality of abnormality types includes utilizing additional labeling data of a subset of the plurality of medical scans, wherein each of the subset of the plurality of medical scans have corresponding labeling data indicating the first one of the plurality of abnormality types is present in the each of the subset of the plurality of medical scans, wherein the additional labeling data indicates characterization labels characterizing the first one of the plurality of abnormality types in the each of the subset of the plurality of medical scans, and wherein the additional abnormality data indicates output data for the characterization labels.

### Claim 18 (depends on 14)

The method of claim 14, further comprising:
determining a second one of the plurality of abnormality types by determining a second corresponding one of the plurality of probability values compares favorably to a detection threshold;
selecting a second one of the plurality of fine-tuned models that corresponds to the second one of the plurality of abnormality types;
generating second additional abnormality data for the new medical scan by performing a second fine-tuned inference function on the image data of the new medical scan, wherein the second fine-tuned inference function utilizes the second one of the plurality of fine-tuned models; and
transmitting the second additional abnormality data to a client device for display via a display device in conjunction with the additional abnormality data.

### Claim 19 (depends on 14)

The method of claim 14, wherein the abnormality detection data indicates region of interest data indicating a location of the first one of the plurality of abnormality types in the image data of the new medical scan, and wherein the method further comprises:
selecting a sub-region of the image data of the new medical scan based on the region of interest data, wherein the fine-tuned inference function is performed only on the sub-region of the image data of the new medical scan.

### Claim 20 (depends on 19)

The method of claim 19, wherein performing the fine-tuning step to generate the one of the plurality of fine-tuned models includes:
determining a subset of the plurality of medical scans that have corresponding labeling data indicating the first one of the plurality of abnormality types is present in the each of the subset of the plurality of medical scans;
selecting a plurality of sub-regions by selecting a sub-region of the image data of each of the subset of the plurality of medical scans based on region interest data of the corresponding labeling data indicating a location of the first one of the plurality of abnormality types in the image data of the each of the subset of the plurality of medical scans; and
performing a training step on the plurality of sub-regions to generate the one of the plurality of fine-tuned models.
