Medical visual question answering
AssigneeInternational Business Machines Corporation
InventorsYuan Zhou, Jing Mei, Shiwan Zhao, Yi Qin Yu, Xu Min, Yan Fei Wang
Life of this patent
Estimating the term…
Abstract
Aspects of the invention include a computer-implemented method including extracting a domain-specific object feature from a first image data, wherein the feature describes an object in the first image data. A domain-specific semantic meaning of text data is determined. The object feature is mapped to a portion of the text data, wherein the portion of the text data describes the object. A joint representation of the object and the portion of the text data is created. A second image data and a query directed towards an object in the second image data is received. An answer to the query is generated based on the joint representation.
Claims20
A computer-implemented method comprising:
- extracting, by a processor, a domain-specific object feature from a first image data, wherein the feature describes an object in the first image data;
- determining, by the processor, domain-specific semantic meaning of text data;
- mapping, by the processor, the object feature to a portion of the text data, wherein the portion of the text data describes the object;
- creating, by the processor, a joint representation of the object and the portion of the text data;
- receiving, by the processor, a second image data and a query directed towards an object in the second image data;
- and
- generating, by the processor, an answer to the query based on the joint representation.
A system comprising:
- a memory having computer readable instructions;
- and
- one or more processors for executing the computer readable instructions, the computer readable instructions controlling the one or more processors to perform operations comprising:
- extracting a domain-specific object feature from a first image data, wherein the feature describes an object in the first image data;
- determining domain-specific semantic meaning of text data;
- mapping the object feature to a portion of the text data, wherein the portion of the text data describes the object;
- creating a joint representation of the object and the portion of the text data;
- receiving a second image data and a query directed towards an object in the second image data;
- and
- generating, by the processor, an answer to the query based on the joint representation.
A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor of a neural network to cause the processor to perform operations comprising:
- extracting a domain-specific object feature from a first image data, wherein the feature describes an object in the first image data;
- determining domain-specific semantic meaning of text data;
- mapping the object feature to a portion of the text data, wherein the portion of the text data describes the object;
- creating a joint representation of the object and the portion of the text data;
- receiving a second image data and a query directed towards an object in the second image data;
- and
- generating, by the processor, an answer to the query based on the joint representation.
Dependencies are inferred from the claim text.