Systems, methods and computer media for joint attention video processing

A computer implemented method for training an action classification model, the method comprising, iteratively, for each video clip in a plurality of video clips of a video:

• receiving a plurality of video frames of the video clip;
• retrieving a plurality of first input features previously extracted based on the plurality of video frames;
• generating a plurality of second input features, by the action classification model, based on the plurality of video frames;
• computing a classification loss of the action classification model based on the plurality of second input features;
• generating an updated set of input features based on the plurality of first input features and the plurality of second input features;
• generating a plurality of action localization labels using an action localization network based on the updated set of input features;
• computing a localization loss of the action localization network based on the plurality of action localization labels;
• computing an updated loss based on the classification loss of the action classification model and the localization loss of the action localization network;
• and
• adjusting one or more weights of the action classification model based on the updated loss.

A system for processing a video, the system comprising:

• a processing unit;
• and
• a memory coupled to the processing unit, the memory storing machine-executable instructions that, when executed by the processing unit, cause the system to:
• receive a plurality of video frames of a video;
• generate a plurality of first input features based on the plurality of video frames;
• generate a plurality of second input features based on reversing a temporal order of the plurality of first input features;
• generate a first set of joint attention features based on the plurality of first input features;
• generate a second set of joint attention features based on the plurality of second input features;
• and
• concatenate the first set of joint attention features and the second set of joint attention features to generate a final set of joint attention features.