Apparatus and methods for supporting medical decisions

A medical decision supporting apparatus comprising:

• a visual and linguistic neural learner configured to receive a medical image and extract visual embedding;
• a cause miner configured to extract a predetermined index based on the visual embedding and pretrained embedding, and output a predetermined cause for the medical image based on a hash map including at least one index-concept pair and the extracted predetermined index;
• a logical reasoner configured to generate a medical report based on the visual embedding, a background knowledge graph, and the predetermined cause;
• and
• a proof inference unit configured to generate an answer including a proof for a user's question about the medical image based on the predetermined cause and the medical report,
• wherein the proof inference unit comprises:
• a visual question answer interpreter configured to receive an input of the user's question about the medical image, extract a type of a visual question and answer, and determine a visual question answer (VQA) triple according to a VQA type;
• a semantic parser configured to receive an input of the VQA triple, the predetermined cause, and the medical report, extract a sentence related to the visual question answer from the predetermined cause and the medical report, and construct a hierarchy in a form of entity, sentence, and context for the extracted sentence;
• and
• a factual reasoning & verifier configured to receive an input of the VQA triple and a duplication-filtered hierarchy in a form of entity, sentence, and context, and generate a proof for the user's question.

A medical decision supporting method performed by a computing device having at least one processor and a memory that stores at least one program executed by the at least one processor, the method comprising:

• a visual and linguistic neural learning step of receiving a medical image and extracting visual embedding;
• a cause mining step of extracting a predetermined index based on the visual embedding and pretrained embedding, and outputting a predetermined cause for the medical image based on a hash map including at least one index-concept pair and the extracted predetermined index;
• a logical reasoning step of generating a medical report based on the visual embedding, a background knowledge graph, and the predetermined cause;
• and
• a proof inference step of generating an answer including a proof for a user's question about the medical image based on the predetermined cause and the medical report,
• wherein the cause mining step comprises:
• receiving an input of the visual embedding and outputting a causal concept;
• storing the visual embedding and the causal concept;
• extracting an index of pretrained embedding that is most similar to input visual embedding among pretrained embedding stored in a causal triple embedding & concept repository by using at least one of cosine similarity or Euclidean similarity;
• matching a concept in the extracted index based on the extracted index and a hash map stored in the causal triple embedding & concept repository;
• and
• transforming the matched concept into a sentence to output an associated causal expression.

A medical decision supporting method performed by a computing device having at least one processor and a memory that stores at least one program executed by the at least one processor, the method comprising:

• a visual and linguistic neural learning step of receiving a medical image and extracting visual embedding;
• a cause mining step of extracting a predetermined index based on the visual embedding and pretrained embedding, and outputting a predetermined cause for the medical image based on a hash map including at least one index-concept pair and the extracted predetermined index;
• a logical reasoning step of generating a medical report based on the visual embedding, a background knowledge graph, and the predetermined cause;
• and
• a proof inference step of generating an answer including a proof for a user's question about the medical image based on the predetermined cause and the medical report,
• wherein the proof inference step comprises:
• receiving an input of the user's question about the medical image, extracting a type of a visual question and answer, and determining a visual question answer (VQA) triple according to a VQA type;
• receiving an input of the VQA triple, the predetermined cause, and the medical report, extracting a sentence related to the visual question answer from the predetermined cause and the medical report, and constructing a hierarchy in a form of entity, sentence, and context for the extracted sentence;
• and
• receiving an input of the VQA triple and a duplication-filtered hierarchy in a form of entity, sentence, and context, and generating a proof for the user's question.

A computer program stored in a non-transitory computer readable storage medium, the computer program comprising one or more instructions that, when executed by a computing device having at least one processor, cause the computing device to execute:

• a visual and linguistic neural learning step of receiving a medical image and extracting visual embedding;
• a cause mining step of extracting a predetermined index based on the visual embedding and pretrained embedding, and outputting a predetermined cause for the medical image based on a hash map including at least one index-concept pair and the extracted predetermined index;
• a logical reasoning step of generating a medical report based on the visual embedding, a background knowledge graph, and the predetermined cause;
• and
• a proof inference step of generating an answer including a proof for a user's question about the medical image based on the predetermined cause and the medical report,
• wherein the cause mining step comprises:
• receiving an input of the visual embedding and outputting a causal concept;
• storing the visual embedding and the causal concept;
• extracting an index of pretrained embedding that is most similar to input visual embedding among pretrained embedding stored in a causal triple embedding & concept repository by using at least one of cosine similarity or Euclidean similarity;
• matching a concept in the extracted index based on the extracted index and a hash map stored in the causal triple embedding & concept repository;
• and
• transforming the matched concept into a sentence to output an associated causal expression.