Spatial-Temporal Reasoning Through Pretrained Language Models for Video-Grounded Dialogues

A system for generating a response for a video grounded dialogue, the method comprising:

• a memory configured to store a video-grounded dialogue neural network language model;
• and
• a processor coupled to the memory and configured to:
• receive, at the video-grounded dialogue neural network language model, a video input and text input, wherein the text input comprises a dialogue history between the video-grounded dialogue neural network language model and a human user, and a current utterance by the human user;
• generate, using a plurality of video encoding layers in the video-grounded dialogue neural network language model, an encoded video input, wherein the encoded video input includes a spatial level encoding and at least one of a modality level encoding, a temporal level encoding, or a position level encoding;
• generate, using a plurality of text encoding layers in the video-grounded dialogue neural network language model, an encoded text input, wherein the encoded text input includes a token level encoding and at least one of a modality level encoding, a turn-level encoding, or a position level encoding;
• concatenate the encoded video input and the encoded text input in to a single input sequence;
• and
• generate from the single input sequence and using a generative pre-trained transformer model in the video-grounded dialogue neural network language model, the response to the current utterance of the human user.

A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations that process a natural language, the operations comprising:

• receiving, at a video-grounded dialogue neural network language model, a video input and text input, wherein the text input comprises a dialogue history between the video-grounded dialogue neural network language model and a human user and a current utterance by the human user;
• generating, using a plurality of video encoding layers in the video-grounded dialogue neural network language model, an encoded video input, wherein the encoded video input includes a spatial level encoding and at least one of a modality level encoding, a temporal level encoding, or a position level encoding;
• generating, using a plurality of text encoding layers in the video-grounded dialogue neural network language model, an encoded text input, wherein the encoded text input includes a token level encoding and at least one of a modality level encoding, a turn-level encoding, or a position level encoding;
• concatenating the encoded video input and the encoded text input in to a single input sequence;
• and
• generating from the single input sequence and using a generative pre-trained transformer model in the video-grounded dialogue neural network language model, the response to the current utterance of the human user.