Unified Vision and Dialogue Transformer with BERT

A system comprising:

• a non-transitory memory;
• and
• one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising:
• receiving, at a visual dialogue module neural network language model, an image input and text input, wherein the text input comprises a dialogue history between the visual dialogue module neural network language model and a human user and a current utterance by the human user;
• generating, using a plurality of image encoding layers in the visual dialogue module neural network language model, an encoded image input, wherein the encoded image input includes a token level encoding and at least one of a position level encoding or a segment level encoding;
• generating, using a plurality of text encoding layers in the visual dialogue module neural network language model, an encoded text input, wherein the encoded text input includes a token level encoding and at least one of a position level encoding or a segment level encoding;
• concatenating the encoded image input and the encoded text input in to a single input sequence;
• and
• generating, from the single input sequence and using a pre-trained language model with one or more self-attention mask layers in the visual dialogue module neural network language model, a response to the current utterance of the human user.

A non-transitory, machine-readable medium having stored thereon machine-readable instructions executable to cause a system to perform operations comprising:

• receiving, at a visual dialogue neural network language model, an image input and text input, wherein the text input comprises a dialogue history between the visual dialogue neural network language model and a human user and a current utterance by the human user;
• generating, from the image input and text input, using a plurality of visual encoding layers in the visual dialogue neural network language model, an encoded visual dialogue input, wherein the encoded visual dialogue input includes a position level encoding and a segment type encoding;
• generating, from the encoded visual dialogue input and using a transformer encoder network in the visual dialogue neural network language model, a unified contextualized representation, wherein the unified contextualized representation includes a token level encoding of the encoded visual dialogue input;
• generating, from the unified contextualized representation and using a first self-attention mask associated with discriminative settings of the transformer encoder network or a second self-attention mask associated with generative settings of the transformer encoder network, an answer prediction;
• and
• providing the answer prediction as a response to the current utterance of the human user.