Method and device for visual question answering, computer apparatus and medium

An apparatus for visual question answering, comprising:

• an acquiring module, configured to acquire an input image and an input question;
• a graph constructing module, configured to construct a Visual Graph based on the input image, wherein the Visual Graph comprises a Node Feature and an Edge Feature;
• an updating module, configured to update the Node Feature by using the Node Feature and the Edge Feature to obtain an updated Visual Graph;
• a question feature extracting module, configured to determine a question feature based on the input question;
• a fusing module, configured to fuse the updated Visual Graph and the question feature to obtain a fused feature;
• and
• a predicting module, configured to generate a predicted answer for the input image and the input question based on the fused feature,
• wherein the updating module is further configured to perform at least one round of updating operation on the Node Feature of the Visual Graph by using a predetermined neural network, wherein the predetermined neural network comprises a Fully Connected Layer, a first Graph Convolutional Layer and a second Graph Convolutional Layer,
• wherein each of the at least one round of updating operation comprises:
• mapping the Node Feature of the Visual Graph to a first feature by using the Fully Connected Layer, wherein a number of spatial dimensions of the first feature equals to a predetermined number;
• processing the first feature by using the first Graph Convolutional Layer to obtain a second feature;
• processing the second feature by using the second Graph Convolutional Layer to obtain the updated Node Feature;
• and
• constructing the updated Visual Graph by the updated Node Feature and the Edge Feature, and
• wherein each of the at least one round of updating operation further comprises:
• constructing a Graph Laplacians based on the Edge Feature;
• and
• wherein the processing the first feature by using the first Graph Convolutional Layer comprises:
• processing the first feature by using the first Graph Convolutional Layer based on the Graph Laplacians to obtain the second feature, wherein the second feature comprises a plurality of first sub-features.

A computer device, comprising:

• a memory having computer instructions stored thereon;
• and
• at least one processor;
• wherein the processor, when executing the computer instructions, is configured to:
• acquire an input image and an input question;
• construct a Visual Graph based on the input image, wherein the Visual Graph comprises a Node Feature and an Edge Feature;
• update the Node Feature by using the Node Feature and the Edge Feature to obtain an updated Visual Graph;
• determine a question feature based on the input question;
• fuse the updated Visual Graph and the question feature to obtain a fused feature;
• and
• generate a predicted answer for the input image and the input question based on the fused feature,
• wherein the processor is further configured to:
• perform at least one round of updating operation on the Node Feature of the Visual Graph by using a predetermined neural network, wherein the predetermined neural network comprises a Fully Connected Layer, a first Graph Convolutional Layer and a second Graph Convolutional Layer,
• wherein each of the at least one round of updating operation comprises:
• mapping the Node Feature of the Visual Graph to a first feature by using the Fully Connected Layer, wherein a number of spatial dimensions of the first feature equals to a predetermined number;
• processing the first feature by using the first Graph Convolutional Layer to obtain a second feature;
• processing the second feature by using the second Graph Convolutional Layer to obtain the updated Node Feature;
• and
• constructing the updated Visual Graph by the updated Node Feature and the Edge Feature, and
• wherein each of the at least one round of updating operation further comprises:
• constructing a Graph Laplacians based on the Edge Feature;
• and
• wherein the processing the first feature by using the first Graph Convolutional Layer comprises:
• processing the first feature by using the first Graph Convolutional Layer based on the Graph Laplacians to obtain the second feature, wherein the second feature comprises a plurality of first sub-features.

A non-transitory computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, are configured to:

• acquire an input image and an input question;
• construct a Visual Graph based on the input image, wherein the Visual Graph comprises a Node Feature and an Edge Feature;
• update the Node Feature by using the Node Feature and the Edge Feature to obtain an updated Visual Graph;
• determine a question feature based on the input question;
• fuse the updated Visual Graph and the question feature to obtain a fused feature;
• and
• generate a predicted answer for the input image and the input question based on the fused feature,
• wherein the computer instructions are further configured to:
• perform at least one round of updating operation on the Node Feature of the Visual Graph by using a predetermined neural network, wherein the predetermined neural network comprises a Fully Connected Layer, a first Graph Convolutional Layer and a second Graph Convolutional Layer,
• wherein each of the at least one round of updating operation comprises:
• mapping the Node Feature of the Visual Graph to a first feature by using the Fully Connected Layer, wherein a number of spatial dimensions of the first feature equals to a predetermined number;
• processing the first feature by using the first Graph Convolutional Layer to obtain a second feature;
• processing the second feature by using the second Graph Convolutional Layer to obtain the updated Node Feature;
• and
• constructing the updated Visual Graph by the updated Node Feature and the Edge Feature, and
• wherein each of the at least one round of updating operation further comprises:
• constructing a Graph Laplacians based on the Edge Feature;
• and
• wherein the processing the first feature by using the first Graph Convolutional Layer comprises:
• processing the first feature by using the first Graph Convolutional Layer based on the Graph Laplacians to obtain the second feature, wherein the second feature comprises a plurality of first sub-features.