Systems and methods for improved video understanding

A computing system configured for classifying video data with improved accuracy, the computing system comprising:

• one or more processors;
• and
• one or more memory devices storing:
• a video understanding model comprising a video transformer encoder model;
• and
• one or more operations that, when implemented by the one or more processors, cause the one or more processors to perform operations, the operations comprising:
• obtaining video data comprising a plurality of video frames;
• extracting a plurality of video tokens from the video data, the plurality of video tokens comprising a representation of spatiotemporal information in the video data,
• wherein extracting the plurality of video tokens from the video data comprises:
• extracting, by the computing system, a plurality of video tubelets from the video data, the plurality of video tublets respectively comprising a length and a width and spanning two or more video frames of the plurality of video frames;
• projecting, by the computing system, the plurality of video tubelets to a plurality of tensor representations of the plurality of video tubelets;
• and
• merging, by the computing system, the plurality of tensor representations along at least one dimension to produce the plurality of video tokens;
• providing the plurality of video tokens as input to a video understanding model, the video understanding model comprising a video transformer encoder model;
• and
• receiving a classification output from the video understanding model.