System and method for self-supervised video transformer

A system for human action recognition in a video, comprising:

• processing circuitry configured to
• sample video clips of a video with varying temporal resolutions in global views, by sampling the video clips at a first frame rate and a second frame rate, wherein the second frame rate is different from the first frame rate, and
• sample the video clips from different spatiotemporal windows in local views;
• machine learning circuitry configured to
• match the global and local views in a framework of student-teacher networks to learn cross-view correspondence between local and global views by performing local-to- global spatiotemporal view matching, and to learn motion correspondence between varying temporal resolutions by performing global-to-global spatiotemporal view matching between the video clips sampled at the first frame rate and at the second frame rate;
• and
• a display device for displaying each video clip in a manner that emphasizes attention to the recognized human action,
• wherein the motion and cross-view correspondences involve varying spatial and temporal resolutions, and the matching includes a dynamic positional encoding of the video clips.

A non-transitory computer readable storage medium storing program code, which when executed by a computer having a CPU and a machine learning engine, perform a method comprising:

• sampling video clips with varying temporal resolutions in global views, by sampling the video clips at a first frame rate and a second frame rate, wherein the second frame rate is different from the first frame rate;
• sampling the video clips from different spatiotemporal windows in local views;
• matching, via the machine learning engine, the global and local views in a framework of student-teacher networks to learn cross-view correspondence between local and global views by performing local-to-global spatiotemporal view matching, and to learn motion correspondence between varying temporal resolutions by performing global-to-global spatiotemporal view matching between the video clips sampled at the first frame rate and at the second frame rate;
• and
• displaying each video clip in a manner that emphasizes attention to the recognized human action,
• wherein the motion and cross-view correspondences involve varying spatial and temporal resolutions, and the matching includes a dynamic positional encoding of the video clips.