# US 11,308,608 B2 · Apparatus and method for medical diagnostic

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,608 B2
- Document ID: US11308608B2
- Status: Granted
- Kind: B2
- Application number: 16/664,832
- Filed: 2019-10-26
- Published: 2021-03-25
- Granted: 2022-04-19
- Earliest claimed: 2019-09-23
- Assignees: Samsung Sds Co., Ltd.; Samsung Life Public Welfare Foundation
- Applicants: Samsung Sds Co., Ltd.; Samsung Life Public Welfare Foundation
- Inventors: Eun-Joo Jeon; Sung-hoon Joo; Soon-Hwan Kwon; Myung-Jin Chung
- Web page: https://patentgrep.com/patent/US11308608B2

## CPC

- G16H 30/40: for processing medical images, e.g. editing
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/24: Classification techniques
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G06T 7/0012: {Biomedical image inspection}
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G06T 2200/04: involving 3D image data
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20212: Image combination
- G06T 2207/30096: Tumor; Lesion
- G06V 2201/03: Recognition of patterns in medical or anatomical images
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A medical diagnostic method according to an embodiment includes receiving a plurality of slide images of a three-dimensional (3D) medical image obtained by capturing an object and clinical data for the object, extracting one or more slide images including a lesion among the plurality of slide images, extracting a region of interest (ROI) including the lesion from each of the extracted one or more slide images, generating latent space data including a feature of an image of the extracted ROI using a pre-trained deep learning-based ROI abstraction model, and generating a diagnostic result for the object from the latent space data and the clinical data using a pre-trained deep learning-based medical diagnostic model.

## Claims

### Claim 1

A medical diagnostic method comprising:
receiving a plurality of slide images of a three-dimensional (3D) medical image obtained by capturing an object and clinical data in addition to the plurality of slide images for the object;
extracting one or more slide images including a lesion among the plurality of slide images;
extracting a region of interest (ROI) including the lesion from each of the extracted one or more slide images;
generating latent space data including a feature of an image of the extracted ROI using a pre-trained deep learning-based ROI abstraction model comprising an encoder that is trained using images of a plurality of ROIs including a lesion as training data; and
generating a diagnostic result for the object from the latent space data and the clinical data using a pre-trained deep learning-based medical diagnostic model,
wherein the medical diagnostic model is trained using a plurality of pieces of clinical data, a diagnostic result for each of pieces of the clinical data, and output data of the ROI abstraction model as training data.

### Claim 2 (depends on 1)

The medical diagnostic method of claim 1, wherein the extracting of one or more slide images comprises extracting the one or more slide images using a convolutional neural network (CNN)-based lesion classification model that is pre-trained using a plurality of pre-collected slide images and a lesion classification result for each of the plurality of pre-collected slide images as training data.

### Claim 3 (depends on 1)

The medical diagnostic method of claim 1, wherein the extracting of an ROI comprises extracting the ROI using a CNN-based ROI extraction model that is pre-trained using a plurality of pre-collected slide images including a lesion and an ROI extraction result for each of the plurality of pre-collected slide images as training data.

### Claim 4 (depends on 1)

The medical diagnostic method of claim 1, wherein the encoder is trained using the images of the plurality of ROIs including the lesion as the training data through variational auto-encoder (VAE).

### Claim 5 (depends on 4)

The medical diagnostic method of claim 4, wherein the generating of latent space data comprises generating the latent space data by using the encoder to encode the image of the extracted ROI.

### Claim 6 (depends on 1)

The medical diagnostic method of claim 1, wherein the ROI abstraction model and the medical diagnostic model are trained such that a sum of a resulting value of a loss function of each of the ROI abstraction model and the medical diagnostic model is minimized.

### Claim 7 (depends on 6)

The medical diagnostic method of claim 6, wherein the sum of the resulting value of the loss function of each of the ROI abstraction model and the medical diagnostic model is calculated using Equation 1 below:
Ltotal=α·VAE_loss+Diagnostic_loss (here, 0<α) [Equation 1]
where Ltotal indicates the sum of the resulting value of the loss function of each of the ROI abstraction model and the medical diagnostic model, VAE_loss indicates a resulting value of a loss function of the ROI abstraction model, and Diagnostic_loss indicates a resulting value of a loss function of the medical diagnostic model.

### Claim 8 (depends on 1)

The medical diagnostic method of claim 1, wherein the diagnostic result comprises one of a survival rate, a treatment status, and a lesion type classification result.

### Claim 9 (depends on 1)

The medical diagnostic method of claim 1, further comprising, after the extracting of an ROI, generating a 3D ROI image obtained by combining the image of the extracted ROI of each of the extracted one or more slide images.

### Claim 10

A medical diagnostic apparatus comprising:
a memory configured to store one or more instructions; and
one or more processors configured to execute the one or more instructions,
wherein the one or more processors:
receive a plurality of slide images of a three-dimensional (3D) medical image obtained by capturing an object and clinical data in addition to the plurality of slide images for the object,
extract one or more slide images including a lesion among the plurality of slide images,
extract a region of interest (ROI) including the lesion from each of the extracted one or more slide images,
generate latent space data in which a feature of an image of the extracted ROI is compressed using a pre-trained deep learning-based ROI abstraction model comprising an encoder that is trained using images of a plurality of ROIs including the lesion as training data, and
generate a diagnostic result for the object from the latent space data and the clinical data using a pre-trained deep learning-based medical diagnostic model,
wherein the medical diagnostic model is trained using a plurality of pieces of clinical data, a diagnostic result for each of pieces of the clinical data, and output data of the ROI abstraction model as training data.

### Claim 11 (depends on 10)

The medical diagnostic apparatus of claim 10, wherein the one or more processors extract the one or more slide images using a convolutional neural network (CNN)-based lesion classification model that is pre-trained using a plurality of pre-collected slide images and a lesion classification result for each of the plurality of pre-collected slide images as training data.

### Claim 12 (depends on 10)

The medical diagnostic apparatus of claim 10, wherein the one or more processors extract the ROI using a CNN-based ROI extraction model that is pre-trained using a plurality of pre-collected slide images including a lesion and an ROI extraction result for each of the plurality of pre-collected slide images as training data.

### Claim 13 (depends on 10)

The medical diagnostic apparatus of claim 10, wherein the encoder is trained using the images of the plurality of ROIs including the lesion as the training data through variational auto-encoder (VAE).

### Claim 14 (depends on 13)

The medical diagnostic apparatus of claim 13, wherein the one or more processors generate the latent space data by using the encoder to encode the image of the extracted ROI.

### Claim 15 (depends on 10)

The medical diagnostic apparatus of claim 10, wherein the ROI abstraction model and the medical diagnostic model are trained such that a sum of a resulting value of a loss function of each of the ROI abstraction model and the medical diagnostic model is minimized.

### Claim 16 (depends on 15)

The medical diagnostic apparatus of claim 15, wherein the sum of the resulting value of the loss function of each of the ROI abstraction model and the medical diagnostic model is calculated using Equation 1 below:
Ltotal=α·VAE_loss+Diagnostic_loss (here, 0<α) [Equation 1]
where Ltotal indicates the sum of the resulting value of the loss function of each of the ROI abstraction model and the medical diagnostic model, VAE_loss indicates a resulting value of a loss function of the ROI abstraction model, and Diagnostic_loss indicates a resulting value of a loss function of the medical diagnostic model.

### Claim 17 (depends on 10)

The medical diagnostic apparatus of claim 10, wherein the diagnostic result comprises one of a survival rate, a treatment status, and a lesion type classification result.

### Claim 18 (depends on 10)

The medical diagnostic apparatus of claim 10, wherein the one or more processors generate a 3D ROI image obtained by combining the image of the extracted ROI of each of the extracted one or more slide images.