Linear attention in vision transformer with quadratic taylor expansion

A method, comprising:

• generating, by a processing circuit, a pair of input vectors representing a pair of input values sent to a machine learning model;
• generating, by the processing circuit, a self-attention score representing an amount of relevance between the pair of input values based on an approximation of a similarity measurement function comprising a product of two terms, wherein at least a first term of the two terms comprises a compact Kronecker product of a first vector of the pair of input vectors;
• and
• conveying, by the processing circuit, the self-attention score to one of a next layer or an output of the machine learning model.

A computing system comprising:

• a memory configured to store an image;
• and
• a processing circuit configured to execute a plurality of layers of a machine learning model in a computation order specified by a computational graph;
• wherein responsive to a request to classify the image, the processing circuit is configured to:
• generate a plurality of input values, each being a non-overlapped patch of the image;
• generate a pair of input vectors representing a pair of input values of the plurality of input values;
• generate a self-attention score representing an amount of relevance between the pair of input values based on an approximation of a similarity measurement function comprising a product of two terms, wherein at least a first term of the two terms comprises a compact Kronecker product of a first vector of the pair of input vectors;
• and
• convey the self-attention score to one of a next layer or an output of the machine learning model.