US 10,949,718 B2Granted

Multi-modal visual question answering system

AssigneeAccenture Global Solutions Limited

InventorsLuca Costabello, Nicholas McCarthy, Rory McGrath, Sumit Pai

Life of this patent

Estimating the term…

Loading patent…

Abstract

The systems and methods described herein may generate multi-modal embeddings with sub-symbolic features and symbolic features. The sub-symbolic embeddings may be generated with computer vision processing. The symbolic features may include mathematical representations of image content, which are enriched with information from background knowledge sources. The system may aggregate the sub-symbolic and symbolic features using aggregation techniques such as concatenation, averaging, summing, and/or maxing. The multi-modal embeddings may be included in a multi-modal embedding model and trained via supervised learning. Once the multi-modal embeddings are trained, the system may generate inferences based on linear algebra operations involving the multi-modal embeddings that are relevant to an inference response to the natural language question and input image.

Claims20

Dependent claims

A method for visual question inference, the method comprising:

receiving an input image and a natural language query;
determining content classifications for portions of the input image;
generating a scene graph for the input image, the scene graph including the content classifications arranged in a graph data structure comprising nodes and edges, the nodes respectively representative of the content classifications for the input image and the edges representative of relationships between the content classifications;
generating multi-modal embeddings based on the input image and the scene graph, the multi-modal embeddings being respectively associated with the nodes, the edges, or any combination thereof, wherein at least a portion of the multi-modal embeddings are generated by:
determining symbolic embeddings for the content classifications of the input image, the symbolic embeddings representative of nodes of the scene graph, edges of the scene graph, or any combination thereof;
determining a sub-symbolic embedding for the input image, the sub-symbolic embedding comprising an image feature vector for the input image;
identifying separate portions of the image feature vector that are representative of the portions of the input image;
generating weighted sub-symbolic embeddings for each of the content classifications by applying weight values to the separate portions of the image feature vector;
aggregating the symbolic embeddings with the weighted sub-symbolic embeddings to form at least the portion of the multi-modal embeddings;
generating a natural language response to the natural language query based on the multi-modal embeddings by:
generating an inference query based on the natural language query, the inference query indicative of the at least one of the content classifications;
selecting, from the multi-modal embeddings, particular multi-modal embeddings associated with at least one of the content classifications;
determining an inference statement based on a distance measurement between the particular multi-modal embeddings;
and
determining the natural language response based on the inference statement;
and
displaying, in response to receipt of the natural language query and the input image, the natural language response.

14 more elements

A system for visual question inference, the system comprising:

a processor, the processor configured to:
receive an input image and a natural language query;
generate a scene graph for the input image, the scene graph comprising content classifications of image data for the input image, the content classifications being arranged in a graph data structure comprising nodes and edges, the nodes respectively representative of the content classifications for the input image and the edges representative of relationships between the content classifications;
determine symbolic embeddings for the input image, the symbolic embeddings representative of nodes of the scene graph, edges of the scene graph, or any combination thereof;
determine a sub-symbolic embeddings for the input image, the sub-symbolic embeddings comprising respective image feature vectors for the input image;
aggregate the symbolic embeddings with the sub-symbolic embeddings to form multi-modal embeddings in a multi-modal embedding space;
identify at least one node and at least one edge in the scene graph based on natural language included in the natural language query, the natural language text being indicative of at least one of the content classifications;
select, from the multi-modal embeddings, particular multi-modal embeddings associated with the at least one of the content classifications;
determine an inference statement based on a distance measurement between the selected multi-modal embeddings;
generate a natural language response based on the inference statement;
and
display the natural language in response on a graphical user interface.

9 more elements

A non-transitory computer readable storage medium comprising computer executable instructions, the instructions executable by a processor to:

receive an input image and a natural language query;
generate a scene graph for the input image, the scene graph comprising content classifications of image data for the input image, the content classifications being arranged in a graph data structure comprising nodes and edges, the nodes respectively representative of the content classifications for the input image and the edges representative of relationships between the content classifications;
determine symbolic embeddings for the input image, the symbolic embeddings representative of nodes of the scene graph, edges of the scene graph, or any combination thereof;
determine a sub-symbolic embeddings for the input image, the sub-symbolic embeddings comprising respective image feature vectors for the input image;
aggregate the symbolic embeddings with the sub-symbolic embeddings to form multi-modal embeddings in a multi-modal embedding space;
identify at least one node and at least one edge in the scene graph based on natural language included in the natural language query, the natural language text being indicative of at least one of the content classifications;
select, from the multi-modal embeddings, particular multi-modal embeddings associated with the at least one of the content classifications;
determine an inference statement based on a distance measurement between the selected multi-modal embeddings;
generate a natural language response based on the inference statement;
and
display the natural language in response on a graphical user interface.

8 more elements

Dependencies are inferred from the claim text.

Facts

Application: 16/406,380Filed 8 May 2019
Published: US 2020/0356829 A112 Nov 2020
Granted: 16 Mar 2021
Term ends: Not estimatedThe source records don’t support an estimate.

Classification9 codes

G06F 40/284
Recognition of textual entities: lexical analysis, e.g. tokenisation or collocates
G06F 3/0484
Interaction techniques based on graphical user interfaces [GUI]: for the control of specific functions or operations, e.g. selecting or…
G06F 18/24
Analysing: classification techniques
G06F 40/30
Handling natural language data: semantic analysis