# US 2021/0343023 A1 · Method and apparatus for automated target and tissue segmentation using multi-modal imaging and ensemble machine learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343023 A1
- Document ID: US20210343023A1
- Status: Published application
- Kind: A1
- Application number: 17/373,036
- Filed: 2021-07-12
- Published: 2021-11-04
- Earliest claimed: 2019-02-07
- Assignees: Vysioneer Inc.
- Applicants: Vysioneer INC.
- Inventors: Jen-Tang Lu
- Web page: https://patentgrep.com/patent/US20210343023A1

## CPC

- G06T 7/174: involving the use of two or more images
- G16H 50/30: for calculating health indices; for individual health risk assessment
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 20/20: Ensemble learning
- G06T 7/0012: {Biomedical image inspection}
- G06T 7/11: Region-based segmentation
- G16H 30/20: for handling medical images, e.g. DICOM, HL7 or PACS
- G16H 30/40: for processing medical images, e.g. editing
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T 2207/20081: Training; Learning
- G06T 2207/20221: Image fusion; Image merging
- G06T 2207/30004: Biomedical image processing
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods and systems for automated target and tissue segmentation using multi-modal imaging and ensemble machine learning models are provided herein. In some embodiments, a method comprises: receiving a plurality of medical images, wherein each of the plurality of medical images includes a target and normal tissue; combining the plurality of medical images to align the target and normal tissue across the plurality of medical images; inputting the combined medical images into each of a plurality of machine learning models; receiving, in response to the input, an output from each of the plurality of machine learning models; combining the results of the plurality of machine learning models; generating a final segmentation image based on the combined results of the plurality of machine learning models; assigning a score to each segmented target and normal tissue; and sorting the segmented targets and normal tissues based on the scores.

## Claims

### Claim 1

A computer-implemented method for automated target and tissue segmentation using at least one medical imaging modality and at least one machine learning model comprising:
receiving at least one medical image modality, wherein at least one medical image of the at least one medical image modalities includes at least one target and/or at least one normal tissue or organ;
inputting the at least one medical image into each of the at least one machine learning models;
receiving, in response to the input, an output from each of the at least one machine learning models, wherein the outputs of the at least one machine learning models are combined when more than one output is received; and
generating a final segmentation image including at least one segmented target and/or at least one normal tissue or organ based on the outputs of the at least one machine learning models.

### Claim 2 (depends on 1)

The method of claim 1, wherein the at least one machine learning model includes at least one of a high-sensitivity network or a high-specificity network.

### Claim 3 (depends on 1)

The method of claim 1, wherein each of the at least one machine learning models is trained separately using different parameters, architectures, image sizes, image resolutions, optimizers, and loss functions.

### Claim 4 (depends on 3)

The method of claim 3, wherein the loss function is related to a volume of a target, tissue and/or organ, wherein a higher loss is assigned to smaller targets, tissues and/or organs and lower loss is assigned to larger targets, tissues and/or organs.

### Claim 5 (depends on 1)

The method of claim 1, wherein each of the at least one machine learning model produces output distinct from the other machine learning models given the same input images.

### Claim 6 (depends on 1)

The method of claim 1, wherein the output from each of the at least one machine learning models is a score map comprising pixel values ranging from 0 to 1.

### Claim 7 (depends on 1)

The method of claim 1, further comprising applying a weight to the output from each of the at least one machine learning models before generating a final segmentation result.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
displaying the final segmentation result, wherein a final score is assigned to each segmented target and/or at least one normal tissue or organ using a linear/nonlinear combination of output score pixels within a segmented region generated from the at least one machine learning model, locations, sizes, and intensities of a segmented range.

### Claim 9 (depends on 8)

The method of claim 8, further comprising displaying a number of most possible targets, tissues and/or organs with a final score for visualization, wherein the number is one of a value received from a user input, a pre-configured value, based on prior knowledge of a number of targets, tissues and/or organs that exist, or an value adjustable via a graphical user interface, wherein the graphical user interface comprises a scroll bar to adjust a sensitivity and/or specificity of the visualization, and, wherein each displayed target, tissue and/or organ is selectable and editable.

### Claim 10

A system for automated target and tissue segmentation using at least one imaging modality and at least one machine learning models comprising:
a) at least one processor;
b) at least one input device; and
c) at least one storage device storing processor-executable instructions which, when executed by the at least one processor, perform a method including:
receiving at least one medical image modality, wherein at least one medical image of the at least one medical image modalities includes at least one target and/or at least one normal tissue or organ;
inputting the at least one medical image into each of the at least one machine learning models;
receiving, in response to the input, an output from each of the at least one machine learning models, wherein the outputs of the at least one machine learning models are combined when more than one output is received; and
generating a final segmentation image including at least one segmented target and/or at least one normal tissue or organ based on the outputs of the at least one machine learning models.

### Claim 11 (depends on 10)

The system of claim 10, wherein the at least one machine learning model includes at least one of a high-sensitivity network or a high-specificity network.

### Claim 12 (depends on 10)

The system of claim 10, wherein each of the at least one machine learning models is trained separately using different parameters, architectures, image sizes, image resolutions, optimizers, and loss functions.

### Claim 13 (depends on 12)

The system of claim 12, wherein the loss function is related to a volume of a target, tissue and/or organ, wherein a higher loss is assigned to smaller targets, tissues and/or organs and lower loss is assigned to larger targets, tissues and/or organs.

### Claim 14 (depends on 10)

The system of claim 10, wherein each of the at least one machine learning model produces output distinct from the other machine learning models given the same input images.

### Claim 15 (depends on 10)

The system of claim 10, wherein the output from each of the at least one machine learning models is a score map comprising pixel values ranging from 0 to 1.

### Claim 16 (depends on 10)

The system of claim 10, further comprising applying a weight to the output from each of the at least one machine learning models before generating a final segmentation result.

### Claim 17 (depends on 16)

The system of claim 16, further comprising:
displaying the final segmentation result, wherein a final score is assigned to each segmented target and/or at least one normal tissue or organ using a linear/nonlinear combination of output score pixels within a segmented region generated from the at least one machine learning model, locations, sizes, and intensities of a segmented range.

### Claim 18 (depends on 17)

The system of claim 17, further comprising:
displaying a number of most possible targets, tissues and/or organs with a final score for visualization, wherein the number is one of a value received from a user input, a pre-configured value, based on prior knowledge of a number of target, tissue and/or organ that exist, or an value adjustable via a graphical user interface, wherein the graphical user interface comprises a scroll bar to adjust a sensitivity and/or specificity of the visualization, and, wherein each displayed target, tissue and/or organ is selectable and editable.

### Claim 19 (depends on 1)

The method of claim 1, further comprising:
assigning a score to each of the at least one segmented target and/or at least one normal tissue or organ; and
sorting the at least one segmented target and/or at least one normal tissue or organ based on the scores.

### Claim 20 (depends on 10)

The system of claim 10, the method further comprising:
assigning a score to each of the at least one segmented target and/or at least one normal tissue or organ; and
sorting the at least one segmented target and/or at least one normal tissue or organ based on the scores.
