# US 12,603,178 B2 · Apparatus and methods for supporting medical decisions

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,603,178 B2
- Document ID: US12603178B2
- Status: Granted
- Kind: B2
- Application number: 18/123,449
- Filed: 2023-03-20
- Published: 2024-05-09
- Granted: 2026-04-14
- Earliest claimed: 2022-11-04
- Assignees: University-Industry Cooperation Group of Kyung Hee University
- Applicants: University-Industry Cooperation Group of Kyung Hee University
- Inventors: Sung Young Lee; Hao Hua-Cam; Ubaid Ur Rehman
- Web page: https://patentgrep.com/patent/US12603178B2

## CPC

- G16H 50/20: for computer-aided diagnosis, e.g. based on medical expert systems
- G16H 50/70: for mining of medical data, e.g. analysing previous cases of other patients
- G06V 10/82: using neural networks
- G06V 30/18: Extraction of features or characteristics of the image
- G16H 15/00: ICT specially adapted for medical reports, e.g. generation or transmission thereof
- G16H 30/40: for processing medical images, e.g. editing
- G16H: HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A medical decision supporting apparatus according to an embodiment may include a visual and linguistic neural learner, which receives a medical image and extracts visual embedding, a cause miner, which extracts a predetermined index based on the visual embedding and pretrained embedding, and outputs a predetermined cause for the medical image based on a hash map including at least one index-concept pair and the extracted predetermined index, a logical reasoner, which generates a medical report based on the visual embedding, a background knowledge graph, and the predetermined cause, and a proof inference unit, which generates an answer including a proof for a user's question about the medical image based on the predetermined cause and the medical report.

## Claims

### Claim 1

A medical decision supporting apparatus comprising:
a visual and linguistic neural learner configured to receive a medical image and extract visual embedding;
a cause miner configured to extract a predetermined index based on the visual embedding and pretrained embedding, and output a predetermined cause for the medical image based on a hash map including at least one index-concept pair and the extracted predetermined index;
a logical reasoner configured to generate a medical report based on the visual embedding, a background knowledge graph, and the predetermined cause; and
a proof inference unit configured to generate an answer including a proof for a user's question about the medical image based on the predetermined cause and the medical report,
wherein the visual and linguistic neural learner comprises:
a visual encoder configured to extract the visual embedding from the medical image;
a text encoder configured to extract text embedding from text data about the medical image; and
a cross-modal decoder configured to extract multimodal embedding based on the visual embedding and the text embedding,
wherein the cause miner comprises:
a modality-agnostic causality decoder configured to receive an input of the visual embedding and output a causal concept;
a causal triple embedding & concept repository configured to store the visual embedding and the causal concept;
a nearest causal candidate retriever configured to extract an index of pretrained embedding that is most similar to input visual embedding among pretrained embedding stored in the causal triple embedding & concept repository by using at least one of cosine similarity or Euclidean similarity; and
a causal concept inference unit configured to match a concept in the extracted index based on the extracted index and the hash map stored in the causal triple embedding & concept repository, and transform the matched concept into a sentence to output an associated causal expression.

### Claim 2 (depends on 1)

The medical decision supporting apparatus of claim 1, wherein the visual encoder comprises at least one convolutional layer including a convolutional kernel, batch normalization layer, and nonlinear activation layer, and a global pooling layer.

### Claim 3 (depends on 1)

The medical decision supporting apparatus of claim 1, wherein the text encoder comprises a word embedding layer, a positional embedding layer, and an addition layer.

### Claim 4 (depends on 1)

The medical decision supporting apparatus of claim 1,
wherein the cross-modal decoder comprises at least one decoder,
wherein each of the at least one decoder outputs the multimodal embedding based on at least one of the visual embedding that is an output from the visual encoder, the text embedding that is an output from the text encoder, or multimodal embedding that is an output form a previous-stage decoder.

### Claim 5 (depends on 1)

The medical decision supporting apparatus of claim 1, wherein the logical reasoner comprises:
a multimodal recognizer configured to generate multimodal embedding based on the visual embedding, text embedding, and graph-based embedding, and generate a diagnosis label based on the multimodal embedding; and
a pathological reasoner configured to generate a logical concept based on the diagnosis label and pathological logic, and generate the medical report by transforming a pathological meta rule generated based on the logical concept and the predetermined cause into a natural language.

### Claim 6

A medical decision supporting apparatus comprising:
a visual and linguistic neural learner configured to receive a medical image and extract visual embedding;
a cause miner configured to extract a predetermined index based on the visual embedding and pretrained embedding, and output a predetermined cause for the medical image based on a hash map including at least one index-concept pair and the extracted predetermined index;
a logical reasoner configured to generate a medical report based on the visual embedding, a background knowledge graph, and the predetermined cause; and
a proof inference unit configured to generate an answer including a proof for a user's question about the medical image based on the predetermined cause and the medical report,
wherein the proof inference unit comprises:
a visual question answer interpreter configured to receive an input of the user's question about the medical image, extract a type of a visual question and answer, and determine a visual question answer (VQA) triple according to a VQA type;
a semantic parser configured to receive an input of the VQA triple, the predetermined cause, and the medical report, extract a sentence related to the visual question answer from the predetermined cause and the medical report, and construct a hierarchy in a form of entity, sentence, and context for the extracted sentence; and
a factual reasoning & verifier configured to receive an input of the VQA triple and a duplication-filtered hierarchy in a form of entity, sentence, and context, and generate a proof for the user's question.

### Claim 7

A medical decision supporting method performed by a computing device having at least one processor and a memory that stores at least one program executed by the at least one processor, the method comprising:
a visual and linguistic neural learning step of receiving a medical image and extracting visual embedding;
a cause mining step of extracting a predetermined index based on the visual embedding and pretrained embedding, and outputting a predetermined cause for the medical image based on a hash map including at least one index-concept pair and the extracted predetermined index;
a logical reasoning step of generating a medical report based on the visual embedding, a background knowledge graph, and the predetermined cause; and
a proof inference step of generating an answer including a proof for a user's question about the medical image based on the predetermined cause and the medical report,
wherein the cause mining step comprises:
receiving an input of the visual embedding and outputting a causal concept;
storing the visual embedding and the causal concept;
extracting an index of pretrained embedding that is most similar to input visual embedding among pretrained embedding stored in a causal triple embedding & concept repository by using at least one of cosine similarity or Euclidean similarity;
matching a concept in the extracted index based on the extracted index and a hash map stored in the causal triple embedding & concept repository; and
transforming the matched concept into a sentence to output an associated causal expression.

### Claim 8 (depends on 7)

The medical decision supporting method of claim 7, wherein the visual and linguistic neural learning step comprises:
extracting the visual embedding from the medical image;
extracting text embedding from text data about the medical image; and
extracting multimodal embedding based on the visual embedding and the text embedding.

### Claim 9 (depends on 8)

The medical decision supporting method of claim 8, wherein in the visual and linguistic neural learning step, a visual encoder is used, which comprises at least one convolutional layer including a convolutional kernel, batch normalization layer, and nonlinear activation layer, and a global pooling layer.

### Claim 10 (depends on 8)

The medical decision supporting method of claim 8, wherein in the visual and linguistic neural learning step, a text encoder is used, which comprises a word embedding layer, a positional embedding layer, and an addition layer.

### Claim 11 (depends on 8)

The medical decision supporting method of claim 8, wherein in the visual and linguistic neural learning step, a cross-modal decoder is used, which comprises at least one decoder, wherein each of the at least one decoder outputs the multimodal embedding based on at least one of the visual embedding that is an output from a visual encoder, the text embedding that is an output from a text encoder, or multimodal embedding that is an output form a previous-stage decoder.

### Claim 12 (depends on 7)

The medical decision supporting method of claim 7, wherein the logical reasoning step comprises:
generating multimodal embedding based on the visual embedding, text embedding, and graph-based embedding, and generating a diagnosis label based on the multimodal embedding; and
generating a logical concept based on the diagnosis label and pathological logic, and generating the medical report by transforming a pathological meta rule generated based on the logical concept and the predetermined cause into a natural language.

### Claim 13

A medical decision supporting method performed by a computing device having at least one processor and a memory that stores at least one program executed by the at least one processor, the method comprising:
a visual and linguistic neural learning step of receiving a medical image and extracting visual embedding;
a cause mining step of extracting a predetermined index based on the visual embedding and pretrained embedding, and outputting a predetermined cause for the medical image based on a hash map including at least one index-concept pair and the extracted predetermined index;
a logical reasoning step of generating a medical report based on the visual embedding, a background knowledge graph, and the predetermined cause; and
a proof inference step of generating an answer including a proof for a user's question about the medical image based on the predetermined cause and the medical report,
wherein the proof inference step comprises:
receiving an input of the user's question about the medical image, extracting a type of a visual question and answer, and determining a visual question answer (VQA) triple according to a VQA type;
receiving an input of the VQA triple, the predetermined cause, and the medical report, extracting a sentence related to the visual question answer from the predetermined cause and the medical report, and constructing a hierarchy in a form of entity, sentence, and context for the extracted sentence; and
receiving an input of the VQA triple and a duplication-filtered hierarchy in a form of entity, sentence, and context, and generating a proof for the user's question.

### Claim 14

A computer program stored in a non-transitory computer readable storage medium, the computer program comprising one or more instructions that, when executed by a computing device having at least one processor, cause the computing device to execute:
a visual and linguistic neural learning step of receiving a medical image and extracting visual embedding;
a cause mining step of extracting a predetermined index based on the visual embedding and pretrained embedding, and outputting a predetermined cause for the medical image based on a hash map including at least one index-concept pair and the extracted predetermined index;
a logical reasoning step of generating a medical report based on the visual embedding, a background knowledge graph, and the predetermined cause; and
a proof inference step of generating an answer including a proof for a user's question about the medical image based on the predetermined cause and the medical report,
wherein the cause mining step comprises:
receiving an input of the visual embedding and outputting a causal concept;
storing the visual embedding and the causal concept;
extracting an index of pretrained embedding that is most similar to input visual embedding among pretrained embedding stored in a causal triple embedding & concept repository by using at least one of cosine similarity or Euclidean similarity;
matching a concept in the extracted index based on the extracted index and a hash map stored in the causal triple embedding & concept repository; and
transforming the matched concept into a sentence to output an associated causal expression.