# US 2026/0093992 A1 · Fast long-context for transformer attention mechanism

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0093992 A1
- Document ID: US20260093992A1
- Status: Published application
- Kind: A1
- Application number: 19/341,362
- Filed: 2025-09-26
- Published: 2026-04-02
- Earliest claimed: 2024-09-27
- Assignees: Royal Bank of Canada
- Applicants: Royal Bank of Canada
- Inventors: Yongchang HAO; Mengyao ZHAI; Hossein HAJIMIRSADEGHI; Sepid HOSSEINI; Frederick TUNG
- Web page: https://patentgrep.com/patent/US20260093992A1

## CPC

- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/045: Combinations of networks
- G06N 3/0475: Generative networks
- G06N 3/096: Transfer learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A fast long-context attention mechanism can be used with any trained transformer model. The best context ranges of tokens for the attention mechanism can be dynamically selected from segments formed from the tokens.

## Claims

### Claim 1

A method for use in determining context keys for an attention mechanism of a transformer model, the method comprising:
generating a plurality of segments, each segment combining a plurality of respective token key mappings generated by mapping the respective token keys using a random feature matrix Ω∈n×d;
generating a token query mapping by mapping the token query using the random feature matrix Ω;
calculating attention scores between the token query mapping and each of the plurality of segments; and
returning the token keys of m segments with the highest attention scores.

### Claim 2 (depends on 1)

The method of claim 1, wherein the random feature matrix is Ω∈n×d, where each element in Ω is sampled from 0,1.

### Claim 3 (depends on 2)

The method of claim 2, wherein each token key mapping is calculated according to:
ϕ
Ω
(
k
)
:=
1
n
⁢
(
exp
⁢
(
ω
1
T
⁢
k
-

k

2
2
)
,
exp
(
ω
2
T
⁢
k
-

k

2
2
)
,
…
,
exp
(
ω
n
T
⁢
k
-

k

2
2
)
)
where:
k∈d is the token key;
ωi∈d is the ith column of Ω.

### Claim 4 (depends on 3)

The method of claim 3, wherein each segment is generated according to:
ϕ
¯
Ω
(
k
j
:
j
+
c
)
:=
1
c
+
1
⁢
∑
l
=
0
c
⁢
ϕ
Ω
(
k
j
+
l
)
where:
the number of key tokens combined in the segment is c+1.

### Claim 5 (depends on 4)

The method of claim 4, wherein the attention score is calculated according to:
a
l
:
l
+
c
:=
ϕ
Ω
T
(
q
)
⁢
ϕ
¯
Ω
(
k
l
:
l
+
c
)
where:
al:l+c is the attention score for the segment of token keys kl to kl+c.

### Claim 6 (depends on 1)

The method of claim 1, wherein each segment is formed from at most c+1 token keys.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
receiving a new token key and new token query;
mapping the new token key and new token query to, respectively, a new token key mapping and a new token query mapping using the random feature matrix;
adding the new token key mapping to a sliding window buffer;
calculating attention scores between the new token query mapping and each of the plurality of segments; and
returning the token keys of m segments with the highest attention scores and the token keys of the token key mappings in the sliding window buffer.

### Claim 8 (depends on 6)

The method of claim 6, further comprising:
receiving a new token key and new token query;
mapping the new token key and new token query to, respectively, a new token key mapping and a new token query mapping using the random feature matrix;
combining the new token key mapping with an existing segment;
calculating attention scores between the new token query mapping and each of the plurality of segments; and
returning the token keys of t segments with the highest attention scores.

### Claim 9 (depends on 7)

The method of claim 7, further comprising:
determining that the plurality of segments should be dynamically restructured;
determining a new segment length (c′) indicating a maximum number of token key mappings combined together in each new segment; and
calculating the new segments according to:
ϕ
¯
Ω
(
k
j
:
j
+
c
′
-
1
)
:=
1
c
′
⁢
∑
l
=
0
c
′
-
1
⁢
ϕ
Ω
(
k
j
+
l
)
.

### Claim 10 (depends on 9)

The method of claim 9, wherein determining that the plurality of segments should be dynamically restructured comprises:
calculating √{square root over (t)};
determining that dynamic restructuring is required when √{square root over (t)}∈N; and
determining that dynamic restructuring is not required otherwise,
where
t is a total number of tokens in a current context.

### Claim 11 (depends on 9)

The method of claim 9, wherein determining that the plurality of segments should be dynamically restructured comprises one or more of:
determining that the number segments has exceeded a threshold; and
determining that a ratio of the number of segments to the segment length has exceeded a threshold.

### Claim 12 (depends on 1)

A system for use in determining context keys for an attention mechanism of a transformer model, the system comprising:
a processor for executing instructions;
a memory storing instructions, which when executed by the processor configure the system to provide a method according to claim 1.

### Claim 13 (depends on 1)

A non-transitory computer readable memory storing instructions, which when executed by a processor of a system configure the system to provide a method according to claim 1.
