Modularized attentive graph networks for fine-grained referring expression comprehension

A computer program product for fine-grained referring expression comprehension, the computer program product comprising one or more computer readable storage media having computer readable program code collectively stored on the one or more computer readable storage media, the computer readable program code being executed by a processor of a computer system to cause the computer system to perform a method comprising:

• receiving a textual expression and an image as inputs;
• and
• executing fine-grained referring expression comprehension, the executing comprising:
• decomposing the textual expression into different textual modules;
• extracting visual regional proposals from the image;
• using language-guided graph neural networks (GNNs) to mine fine-grained object relations from the visual regional proposals;
• and
• aggregating different matching similarities between the different textual modules and the fine-grained object relations,
• wherein the decomposing of the textual expression comprises modularized language parsing of the textual expression into main, spatial and relation modules, the modularized language parsing comprising:
• using one or more language parser tools to parse a language of the textual expression into subphrases and average word embeddings of each subphrase to obtain a text representation for each of the main module, the spatial module and the relation module, and
• representing each word in the textual expression with word embeddings and adopting a sequence model to encode the textual expression and to aggregate feature vectors and a representative for each of the main module, the spatial module and the relation module, and
• wherein the extracting of the visual regional proposals from the image comprises region proposal networking to detect objects in the image, the visual regional proposals being extracted for the image and used to obtain a set of object proposal candidates for the image and building graphs to represent each object in the image with each object in the image regarded as a node in the graphs and relationships between two objects are regarded as edges between two nodes and with the text representations guiding message passing and information flows of nodes in the graphs.

A computing system, comprising:

• a processor;
• a memory coupled to the processor;
• and
• one or more computer readable storage media coupled to the processor, the one or more computer readable storage media collectively containing instructions that are executed by the processor via the memory to implement a method comprising:
• receiving a textual expression and an image as inputs;
• and
• executing fine-grained referring expression comprehension, the executing comprising:
• decomposing the textual expression into different textual modules;
• extracting visual regional proposals from the image;
• using language-guided graph neural networks (GNNs) to mine fine-grained object relations from the visual regional proposals;
• and
• aggregating different matching similarities between the different textual modules and the fine-grained object relations,
• wherein the decomposing of the textual expression comprises modularized language parsing of the textual expression into main, spatial and relation modules, the modularized language parsing comprising:
• using one or more language parser tools to parse a language of the textual expression into subphrases and average word embeddings of each subphrase to obtain a text representation for each of the main module, the spatial module and the relation module, and
• representing each word in the textual expression with word embeddings and adopting a sequence model to encode the textual expression and to aggregate feature vectors and a representative for each of the main module, the spatial module and the relation module, and
• wherein the extracting of the visual regional proposals from the image comprises region proposal networking to detect objects in the image, the visual regional proposals being extracted for the image and used to obtain a set of object proposal candidates for the image and building graphs to represent each object in the image with each object in the image regarded as a node in the graphs and relationships between two objects are regarded as edges between two nodes and with the text representations guiding message passing and information flows of nodes in the graphs.