Systems and methods for visual question answering
AssigneeSalesforce, Inc.
InventorsAnthony Meng Huat Tiong, Junnan Li, Chu Hong Hoi
Life of this patent
Estimating the term…
Abstract
Embodiments described herein provide a zero-shot visual question answering (VQA) framework, which conjoins foundation network models with zero additional training. A first image and a question relating to the first image are received. The first image is divided into a plurality of image patches. A plurality of relevant image patches that are relevant to the question are determined, using a first neural network model, from the plurality of image patches. A plurality of image captions are generated, using a second neural network model, based on the plurality of relevant image patches. An answer to the question is generated based on the plurality of image captions.
Claims17
A method of zero-shot visual question answering, the method comprising:
- receiving, via a data interface, a first image and a question relating to the first image;
- dividing the first image into a plurality of image patches;
- determining, using a first neural network model, relevance of each image patch of the plurality of image patches to the question;
- generating, using a second neural network model, a plurality of image captions based on the relevance of each image patch to the question, wherein the generating the plurality of image captions includes:
- sampling the plurality of image patches to generate a plurality of sampled image patches based on the relevance of each image patch;
- and
- generating an image caption of the plurality of image captions based on the plurality of sampled image patches;
- and
- generating, using a third neural network model, an answer in response to an input of the question and the plurality of image captions.
A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform a method comprising:
- receiving, via a data interface, a first image and a question relating to the first image;
- dividing the first image into a plurality of image patches;
- determining, using a first neural network model, relevance of each image patch of the plurality of image patches to the question;
- generating, using a second neural network model, a plurality of image captions based on the relevance of each image patch to the question;
- and
- generating, using a third neural network model, an answer in response to an input of the question and the plurality of image captions, wherein the generating the answer further comprises:
- for each image caption, concatenating the image caption with the question to generate a question and caption combination of a plurality of question and caption combinations;
- encoding each of the plurality of question and caption combinations to generate a plurality of corresponding encoded representation;
- combining the plurality of encoded representations to provide a concatenated encoded representation;
- and
- generating the answer by decoding the concatenated encoded representation.
A system, comprising:
- a non-transitory memory;
- and
- one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform a method comprising:
- receiving, via a data interface, a first image and a question relating to the first image;
- dividing the first image into a plurality of image patches;
- determining, using a first neural network model, relevance of each image patch of the plurality of image patches to the question;
- generating, using a second neural network model, a plurality of image captions based on the relevance of each image patch to the question, wherein the generating the plurality of image captions includes:
- sampling the plurality of image patches to generate a plurality of sampled image patches based on the relevance of each image patch;
- and
- generating an image caption of the plurality of image captions based on the plurality of sampled image patches;
- and
- generating, using a third neural network model, an answer in response to an input of the question and the plurality of image captions.
Dependencies are inferred from the claim text.