Visual question answering using model trained on unlabeled videos
AssigneeInternational Business Machines Corporation
InventorsElad Amrani, Rami Ben-Ari, Daniel Nechemia Rotman, Udi Barzelay
Life of this patent
Estimating the term…
Abstract
An example system includes a processor to learn a shared embedding space on unlabeled videos using speech visual correspondence. The processor can learn a number of additional embeddings including a question plus video embedding and an answer embedding using the shared embedding space to generate a trained visual question answering model. The processor can execute a visual question answering based on the trained visual question answering model.
Claims20
A system, comprising a processor to:
- learn a shared embedding space on unlabeled videos using a speech visual correspondence;
- learn a plurality of additional embeddings comprising a question plus video embedding and an answer embedding using the shared embedding space to generate a trained visual question answering model;
- and
- execute a visual question answering based on the trained visual question answering model.
A computer-implemented method, comprising:
- learning, via a processor, a shared embedding space on unlabeled videos using a speech visual correspondence;
- learning, via the processor, a plurality of additional embeddings comprising a question plus video embedding and an answer embedding using the shared embedding space to generate a trained visual question answering model;
- and
- executing, via the processor, a visual question answering based on the trained visual question answering model.
A computer program product for training visual question answering models, the computer program product comprising a computer-readable storage medium having program code embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, the program code executable by a processor to cause the processor to:
- learn a shared embedding space on unlabeled videos using a speech visual correspondence;
- learn a plurality of additional embeddings comprising a question plus video embedding and an answer embedding using the shared embedding space to generate a trained visual question answering model;
- and
- execute a visual question answering based on the trained visual question answering model.
Dependencies are inferred from the claim text.