Spatio-temporal interaction network for learning object interactions

A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for improving video understanding tasks based on higher-order object interactions (HOIs) between object features comprising:

• obtaining a plurality of frames of a video;
• generating a coarse-grained feature representation, including generating an image feature for each of a plurality of timesteps respectively corresponding to each of the frames, and performing attention based on the image features;
• generating a fine-grained feature representation, including generating an object feature for each of the plurality of timesteps, and generating the HOIs between the object features;
• and
• concatenating the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation.

A system for improving video understanding tasks based on higher-order object interactions (HOIs) between object features, comprising:

• at least one processor operatively coupled to a memory, wherein the at least one processor is configured to:
• obtain a plurality of frames of a video;
• generate a coarse-grained feature representation by generating an image feature for each of a plurality of timesteps respectively corresponding to each of the frames, and performing attention based on the image features;
• generate a fine-grained feature representation by generating an object feature for each of the plurality of timesteps, and generating the HOIs between the object features;
• and
• concatenate the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation.