Multi-hop transformer for spatio-temporal reasoning and localization

A non-transitory computer-readable storage medium comprising a computer-readable program for using a multi-hop reasoning framework to perform multi-step compositional long-term reasoning, wherein the computer-readable program when executed on a computer causes the computer to perform the steps of:

• extracting feature maps and frame-level representations from a video stream by using a convolutional neural network (CNN);
• performing object representation learning and detection;
• linking objects through time via tracking to generate object tracks and image feature tracks;
• feeding the object tracks and the image feature tracks to a multi-hop transformer that hops over frames in the video stream while concurrently attending to one or more of the objects in the video stream until the multi-hop transformer arrives at a correct answer;
• and
• employing video representation learning and recognition from the objects and image context to locate a target object within the video stream.

A system for using a multi-hop reasoning framework to perform multi-step compositional long-term reasoning, the system comprising:

• a memory;
• and
• one or more processors in communication with the memory configured to:
• extract feature maps and frame-level representations from a video stream by using a convolutional neural network (CNN);
• perform object representation learning and detection;
• link objects through time via tracking to generate object tracks and image feature tracks;
• feed the object tracks and the image feature tracks to a multi-hop transformer that hops over frames in the video stream while concurrently attending to one or more of the objects in the video stream until the multi-hop transformer arrives at a correct answer;
• and
• employ video representation learning and recognition from the objects and image context to locate a target object within the video stream.