Long sequence modeling via state space model (ssm)-enhanced transformer

A computerized method, comprising:

• for each of a plurality of local input sequences in a global input sequence, receiving, at a global layer of a transformer, tokenized embeddings for each of a plurality of tokens in the local input sequence, from an embedding layer;
• computing, at the global layer, a global self-attention vector for each of the tokenized embeddings in the local input sequence;
• receiving, at a local layer, the global self-attention vector for each local input sequence from the global layer;
• computing, at the local layer, local self-attention for the local input sequence;
• adding and normalizing the global self-attention vector with the local self-attention vector to thereby produce an encoder representation including a self-attention vector for each local input sequence that includes both global self-attention values and local self-attention values;
• and
• outputting a prediction for the global input sequence according to a prediction task based on the encoder representation of each of the local input sequences of the global input sequence.

A computing device, comprising:

• a transformer including an encoder having
• a global layer configured to:
• for each of a plurality of local input sequences in a global input sequence, receive tokenized embeddings for each of a plurality of tokens in the local input sequence, from an embedding layer;
• compute a global self-attention vector for each of the tokenized embeddings in the local input sequence;
• a local layer configured to
• receive the global self-attention vector for each local input sequence from the global layer and compute local self-attention for the local input sequence;
• and
• add and normalize the global self-attention vector with the local self-attention vector to thereby produce an encoder representation including a self-attention vector for each local input sequence that includes both global self-attention values and local self-attention values, wherein
• the transformer is configured to output a prediction for the global input sequence according to a prediction task based on the encoder representation of each of the local input sequences of the global input sequence,
• the global layer includes a state space model layer configured to receive the tokenized embeddings for each of the plurality of tokens in the local input sequence, from the embedding layer,
• the state space model layer includes a discrete time structured state space sequence model parameterized by normal plus low rank matrices, and
• computation of the global self-attention using the global layer including the state space model layer is accomplished with linear computational complexity and linear memory complexity relative to the global input sequence.