Method and system for visio-linguistic understanding using contextual language model reasoners

A system, comprising:

• a memory storing instructions;
• one or more communication interfaces;
• and
• one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
• receive an image and a text corresponding to the image, wherein the image comprises one or more embedded texts and wherein the image and the text correspond to a downstream task;
• convert using a first deep learning model, the image into (i) one or more image captions corresponding to the image (ii) a set of bounding box coordinate tuples, wherein each bounding box coordinate tuple from the set of bounding box coordinate tuples corresponds to one of the one or more image captions and (iii) a set of confidence scores wherein each confidence score from the set of confidence scores corresponds to one of the one or more image captions;
• extract one or more extracted texts from the one or more embedded texts in the image using a second deep learning model;
• convert using a neural network, the set of bounding box coordinate tuples into a set of positional embeddings;
• order the one or more image captions based on (i) the set of confidence scores and (ii) a first set of predefined rules to obtain an ordered set of image captions;
• obtain a set of ordered external knowledge texts using (i) the one or more image captions, (ii) the text and (iii) an external knowledge graph;
• convert the one or more extracted texts, the ordered set of image captions and the set of ordered external knowledge texts into a combined input, wherein the combined input is in a format compatible with a contextual language model reasoner;
• and
• obtain using the contextual language model reasoner, a set of contextual embeddings corresponding to the downstream task from the set of positional embeddings and the combined input.

One or more non-transitory machine readable information storage mediums comprising one or more instructions, which when executed by one or more hardware processors causes a method for:

• receiving an image and a text corresponding to the image, wherein the image comprises one or more embedded texts and wherein the image and the text correspond to a downstream task;
• converting using a first deep learning model, the image into (i) one or more image captions corresponding to the image (ii) a set of bounding box coordinate tuples, wherein each bounding box coordinate tuple from the set of bounding box coordinate tuples corresponds to one of the one or more image captions and (iii) a set of confidence scores wherein each confidence score from the set of confidence scores corresponds to one of the one or more image captions;
• extracting one or more extracted texts from the one or more embedded texts in the image using a second deep learning model;
• converting using a neural network, the set of bounding box coordinate tuples into a set of positional embeddings;
• ordering the one or more image captions based on (i) the set of confidence scores and (ii) a first set of predefined rules to obtain an ordered set of image captions;
• obtaining a set of ordered external knowledge texts using (i) the one or more image captions, (ii) the text (iii) the one or more extracted texts and (iv) an external knowledge graph;
• converting the one or more extracted texts, the ordered set of image captions and the set of ordered external knowledge texts into a combined input, wherein the combined input is in a format compatible with a contextual language model reasoner;
• and
• obtaining using the contextual language model reasoner, a set of contextual embeddings corresponding to the downstream task from the set of positional embeddings and the combined input.