Vision transformers leveraging temporal redundancy

A system for processing frame data, the system comprising:

• a processor;
• and
• a memory having stored thereon a set of instructions which, when executed by the processor, cause the processor to:
• obtain a sequence of data, the sequence comprising at least a first frame;
• apply the sequence of data to a machine learning model, the machine learning model configured to:
• determine at least one token-level error based on a difference between at least one token of a first representation, corresponding to the first frame, and a second representation, the first representation comprising a plurality of first tokens;
• select a token subset of the first representation based on the at least one token-level error;
• perform a first transformer operation based on the token subset to produce an output representation corresponding to the token subset;
• scatter the output representation to a buffer representation corresponding to the first representation;
• and
• produce a machine learning output score based on the buffer representation;
• and
• provide an inference result based on the machine learning output score from the machine learning model.