# US 2025/0095349 A1 · Vision transformers leveraging temporal redundancy

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095349 A1
- Document ID: US20250095349A1
- Status: Published application
- Kind: A1
- Application number: 18/469,418
- Filed: 2023-09-18
- Published: 2025-03-20
- Earliest claimed: 2023-09-18
- Assignees: Wisconsin Alumni Research Foundation
- Applicants: Wisconsin Alumni Research Foundation
- Inventors: Matthew Dutson; Yin Li; Mohit Gupta
- Web page: https://patentgrep.com/patent/US20250095349A1

## CPC

- G06V 10/82: using neural networks
- G06V 10/86: using syntactic or structural representations of the image or video pattern, e.g. symbolic string recognition; using graph matching
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Methods and systems for processing frame data are disclosed. The methods and systems include: obtaining a sequence of data, the sequence comprising at least a first frame; applying the sequence of data to a machine learning model; and providing an inference result based on the machine learning output score from the machine learning model. The machine learning model is configured to: determine at least one token-level error based on a difference between at least one token of a first representation corresponding to the first frame and a second representation; perform a first transformer operation based on the token subset to produce an output representation corresponding to the token subset; scatter the output representation to a buffer representation corresponding to the first representation; and produce a machine learning output score based on the buffer representation. Other aspects, embodiments, and features are also claimed and described.

## Claims

### Claim 1

A method for processing frame data comprising:
obtaining a sequence of data, the sequence comprising at least a first frame;
applying the sequence of data to a machine learning model, the machine learning model configured to:
determine at least one token-level error based on a difference between at least one token of a first representation, corresponding to the first frame, and a second representation, the first representation comprising a plurality of first tokens;
select a token subset of the first representation based on the at least one token-level error;
perform a first transformer operation based on the token subset to produce an output representation corresponding to the token subset;
scatter the output representation to a buffer representation corresponding to the first representation; and
produce a machine learning output score based on the buffer representation; and
providing an inference result based on the machine learning output score from the machine learning model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the machine learning model comprises a vision transformer deep learning model.

### Claim 3 (depends on 1)

The method of claim 1, wherein after determining the at least one token-level error, the machine learning model is configured to update a reference representation based on the token-level error.

### Claim 4 (depends on 1)

The method of claim 1, wherein the second representation comprises a reference representation, and
wherein the machine learning model is configured to determine the token-level error by the difference between corresponding tokens of the first representation and the reference representation.

### Claim 5 (depends on 1)

The method of claim 1, wherein the sequence of data further comprises a second frame, the first frame being subsequent to the second frame in time order,
wherein the second representation corresponds to the second frame,
wherein the machine learning model is configured to determine the token-level error by adding corresponding values from a residual error representation to the difference between corresponding tokens of the first representation and the second representation, and
wherein the residual error representation comprises an accumulated error representation of previous frames of the sequence of data.

### Claim 6 (depends on 1)

The method of claim 1, wherein the machine learning model is further configured to generate a binary mask based on one or more token-level errors that exceed a threshold,
wherein the machine learning model is configured to select the token subset based on the binary mask, and
wherein the machine learning model is configured to update a reference representation according to the binary mask.

### Claim 7 (depends on 1)

The method of claim 1, wherein the machine learning model is further configured to select the token subset based on a predetermined number of tokens having largest token-level errors.

### Claim 8 (depends on 1)

The method of claim 1, wherein the machine learning model is further configured to select the token subset based on a predetermined threshold for token-level error.

### Claim 9 (depends on 1)

The method of claim 1, wherein the machine learning model is further configured to generate a binary mask based on the token-level error, and
wherein the machine learning model is configured to select the token subset by multiplying the first representation and the binary mask.

### Claim 10 (depends on 1)

The method of claim 1, wherein to scatter the output representation, the machine learning model is configured to place a plurality of output tokens of the output representation into the buffer representation to correspond to a plurality of locations in the buffer representation.

### Claim 11 (depends on 1)

The method of claim 1, wherein the first transformer operation comprises a linear transform to produce the output representation, the output representation comprising a query output representation, a key output representation, and a value output representation, each of the query output representation, the key output representation, and the value output representation corresponding to the token subset, and
wherein the buffer representation comprises a query buffer representation scattered from the query output representation, a key buffer representation corresponding the key output representation, and a value buffer representation corresponding to the value output representation.

### Claim 12 (depends on 11)

The method of claim 11, wherein the machine learning model is further configured to:
transpose the key output representation and the key buffer representation; and
generate a query-key representation based on the query output representation, the query buffer representation, the transposed key output representation, and the transposed key buffer representation.

### Claim 13 (depends on 12)

The method of claim 12, wherein to generate the query-key representation, the machine learning model is configured to:
multiply the query buffer representation and the transposed key output representation to produce a first product representation;
update the first product representation by scattering the first product representation into the query-key representation;
multiply the query output representation and the transposed key buffer representation to produce a second product representation; and
update the second product representation by scattering the second product representation into the query-key representation.

### Claim 14 (depends on 12)

The method of claim 12, wherein the machine learning model is further configured to:
convert the query-key representation into a probability representation; and
produce an attention-value representation based on a previous probability representation, a previous value buffer representation, a first change in the probability representation, and a second change in the value buffer representation.

### Claim 15 (depends on 1)

The method of claim 1, wherein the first transformer operation comprises a multilayer perceptron.

### Claim 16

A system for processing frame data, the system comprising:
a processor; and
a memory having stored thereon a set of instructions which, when executed by the processor, cause the processor to:
obtain a sequence of data, the sequence comprising at least a first frame;
apply the sequence of data to a machine learning model, the machine learning model configured to:
determine at least one token-level error based on a difference between at least one token of a first representation, corresponding to the first frame, and a second representation, the first representation comprising a plurality of first tokens;
select a token subset of the first representation based on the at least one token-level error;
perform a first transformer operation based on the token subset to produce an output representation corresponding to the token subset;
scatter the output representation to a buffer representation corresponding to the first representation; and
produce a machine learning output score based on the buffer representation; and
provide an inference result based on the machine learning output score from the machine learning model.

### Claim 17 (depends on 16)

The system of claim 16, wherein the second representation comprises a reference representation, and
wherein the machine learning model is configured to determine the token-level error by the difference between corresponding tokens of the first representation and the reference representation.

### Claim 18 (depends on 16)

The system of claim 16, wherein the sequence of data further comprises a second frame, the first frame being subsequent to the second frame in time order,
wherein the second representation corresponds to the second frame,
wherein the machine learning model is configured to determine the token-level error by adding corresponding values from a residual error representation to the difference between corresponding tokens of the first representation and the second representation, and
wherein the residual error representation comprises an accumulated error representation of previous frames of the sequence of data.

### Claim 19 (depends on 16)

The system of claim 16, wherein the machine learning model is further configured to generate a binary mask based on one or more token-level errors that exceed a threshold,
wherein the machine learning model is configured to select the token subset based on the binary mask, and
wherein the machine learning model is configured to update a reference representation according to the binary mask.

### Claim 20 (depends on 16)

The system of claim 16, wherein the first transformer operation comprises a linear transform to produce the output representation, the output representation comprising a query output representation, a key output representation, and a value output representation, each of the query output representation, the key output representation, and the value output representation corresponding to the token subset, and
wherein the buffer representation comprises a query buffer representation scattered from the query output representation, a key buffer representation corresponding the key output representation, and a value buffer representation corresponding to the value output representation.
