Method of and system for explainable knowledge-based visual question answering

A method for training a visual question answering machine learning model, the method being executed by a processor, the processor having access to:

• a first machine learning model having been trained to generate scene graph of images based on features thereof,
• a second machine learning model having been trained to generate word embeddings, and
• the visual question answering machine learning model, the method comprising:
• obtaining, via the first machine learning model, a scene graph of an image, the scene graph comprising a set of objects having been labelled in the image, the scene graph encoding
• semantic information and spatial information of each labelled object of the set of labelled objects in the image, and
• at least one type of relation between at least two labelled objects of the set of labelled objects;
• obtaining, from a knowledge graph, based on at least a portion of the scene graph, a set of additional objects;
• embedding, using the first machine learning model, the obtained set of additional objects to obtain a set of additional embedded objects;
• and
• generating, using the first machine learning model, the augmented scene graph by enhancing the scene graph with the set of additional embedded objects, the augmented scene graph thereby encoding at least one additional type of relation between at least one labelled object of the set of labelled object and at least one additional object of the set of additional objects;
• obtaining, via the second machine learning model, a set of embedded questions and a set of associated embedded answers, the set of embedded questions and the set of associated embedded answers having been generated from a set of questions and a set of associated answers related to the image;
• and
• training the visual question answering machine learning model to provide an answer in response to a given question related to the image based on:
• the augmented scene graph,
• the set of embedded questions, and
• the set of associated embedded answers.

A system for generating an augmented scene graph of an image, the system comprising a processor, the processor having access to:

• a machine learning model having been trained to generate scene graph of images based on features thereof,
• the processor being operatively connected to a non-transitory storage medium comprising instructions,
• the processor, upon executing the instructions, being configured for:
• obtaining, via the machine learning model, a scene graph of an image, the scene graph comprising a set of objects having been labelled in the image, the scene graph encoding:
• semantic features and spatial features of each labelled object of the set of labelled objects in the image, and
• at least one first type of relation between at least two labelled objects of the set of labelled objects;
• obtaining, from a knowledge graph, based on at least a portion of the scene graph, a set of additional objects;
• embedding, using the machine learning model, the obtained set of additional objects to obtain a set of additional embedded objects;
• and
• generating, using the machine learning model, the augmented scene graph by enhancing the scene graph with the set of additional embedded objects, the augmented scene graph thereby encoding at least one additional type of relation between at least one labelled object of the set of labelled objects and at least one additional object of the set of additional objects, the at least one additional type of relation being a second type of relation.