# US 2025/0053821 A1 · Auto-regressive system and Auto-regressive Method for a Large Language Model

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0053821 A1
- Document ID: US20250053821A1
- Status: Published application
- Kind: A1
- Application number: 18/769,443
- Filed: 2024-07-11
- Published: 2025-02-13
- Earliest claimed: 2023-08-11
- Assignees: MediaTek Singapore Pte. Ltd.
- Applicants: MediaTek Singapore Pte. Ltd.
- Inventors: Jia Yao Christopher LIM; Kelvin Kae Wen TEH; Po-Yen LIN; Jung Hau FOO; Chia-Wei HSU; Yu-Lung LU; Hung-Jen CHEN; Chung-Li LU; Wai Mun WONG
- Web page: https://patentgrep.com/patent/US20250053821A1

## CPC

- G06N 3/091: Active learning
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/08: Learning methods
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An auto-regressive method for a large language model includes receiving a hidden state associated with at least one token, generating key data, first value data, and query data according to a received hidden state, generating first positionally encoded key data by encoding the key data positionally, generating positionally encoded query data by encoding the query data positionally, performing first element-wise dot product operations according to the first positionally encoded key data, the positionally encoded query data, and second positionally encoded key data to generate an attention score, performing second element-wise dot product operations according to the first value data, the attention score, and second value data to generate an attention output, and adding the attention output and the hidden state to generate an updated hidden output.

## Claims

### Claim 1

An auto-regressive method for a transformer-based large language model (LLM), the auto-regressive method comprising:
receiving a hidden state associated with at least one token;
generating key data, first value data, and query data according to a received hidden state;
generating first positionally encoded key data by encoding the key data positionally;
generating positionally encoded query data by encoding the query data positionally;
performing first element-wise dot product operations according to the first positionally encoded key data, the positionally encoded query data, and second positionally encoded key data to generate an attention score;
performing second element-wise dot product operations according to the first value data, the attention score, and second value data to generate an attention output; and
adding the attention output and the hidden state to generate an updated hidden output;
wherein the second positionally encoded key data is obtained and cached before the first positionally encoded key data is generated, and the second value data is obtained and cached before the first value data is generated.

### Claim 2 (depends on 1)

The method of claim 1, wherein performing first element-wise dot product operations according to the first positionally encoded key data, the positionally encoded query data, and the second positionally encoded key data to generate an attention score comprises:
performing a first matrix dot product on the first positionally encoded key data and the positionally encoded query data to generate a first product output;
performing a second matrix dot product on the second positionally encoded key data and the positionally encoded query data to generate a second product output; and
concatenating the first product output and the second product output to generate the attention score.

### Claim 3 (depends on 2)

The method of claim 2, wherein performing second element-wise dot product operations according to the first value data, the attention score, and second value data to generate an attention output comprises:
performing a third matrix dot product according to the attention score and the second value data to generate a third product output;
performing a fourth matrix dot product according to the attention score and the first value data to generate a fourth product output; and
adding the third product output and the fourth product output to generate the attention output.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
performing a softmax function to the attention score for generating an attention weight, wherein the second element-wise dot product operations are performed according to the first value data, the attention weight, and the second value data to generate the attention output.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
normalizing the hidden state as a normalized hidden state by adjusting a square sum of elements of the hidden state equal to a fixed number, wherein the key data, the first value data, and the query data are generated according to the normalized hidden state; and
normalizing the updated hidden output as a normalized updated hidden output by adjusting a square sum of elements of the updated hidden output equal to the fixed number.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
slicing a part of memory segments from a ring buffer to generate a first value cache and/or a first key cache;
slicing another part of memory segments from the ring buffer as sliding windows to generate a second value cache and/or a second key cache;
wherein the first value cache is configured to save the first value data, the second value cache is configured to save the second value data, the first key cache is configured to save the first positionally encoded key data, and the second key cache is configured to save the second positionally encoded key data.

### Claim 7 (depends on 6)

The method of claim 6, further comprising:
updating the part of memory segments from the ring buffer by incrementing an offset value of memory addresses after the first value cache and/or a first key cache is saved;
wherein after the part of memory segments from the ring buffer are updated, the part of memory segments comprises the first value cache and/or a first key cache, and a part of second positionally encoded key data and/or a part of second value data, and the second value data and/or the second positionally encoded key data are updated after the part of memory segments are shifted.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
copying the part of memory segments of the ring buffer according to their initialized memory addresses after the part of memory segments are shifted to hit an end memory segment of the ring buffer.

### Claim 9 (depends on 1)

The method of claim 1, wherein a signal shape format of the first value data and the first positionally encoded key data is represented as (B, N, T, D), a signal shape format of the second value data and the second positionally encoded key data is represented as (B, N, C, D), wherein B is a batch size, T is a token amount, N is an attention head amount, D is a head dimension for each attention head, and C is a used-defined number previously determined.

### Claim 10 (depends on 4)

The method of claim 4, further comprising:
replacing null values of the attention score with mask values to generate an updated attention score.

### Claim 11

An auto-regressive system for a transformer-based large language model (LLM), the auto-regressive system comprising:
a layer input module, configured to receive hidden state associated with at least one token processed by the LLM;
a linear transformation module, configured to generate key data, first value data, and query data by performing linear transformations according to the hidden state received by the layer input module;
a key positional encoder, configured to generate first positionally encoded key data by encoding the key data positionally;
a query positional encoder, configured to generate positionally encoded query data by encoding the query data positionally;
a first multiplication module, configured to perform first element-wise dot product operations according to the first positionally encoded key data, the positionally encoded query data, and second positionally encoded key data to generate an attention score;
a second multiplication module, configured to perform second element-wise dot product operations according to the first value data, the attention score, and second value data to generate an attention output; and
a first adder, configured to add the attention output and the hidden state to generate an updated hidden output;
wherein the second positionally encoded key data is obtained and cached before the first positionally encoded key data is generated, and the second value data is obtained and cached before the first value data is generated.

### Claim 12 (depends on 11)

The system of claim 11, wherein the first multiplication module comprises:
a first batch matrix multiplication module, configured to perform a first matrix dot product on the first positionally encoded key data and the positionally encoded query data to generate a first product output;
a second batch matrix multiplication module, configured to perform a second matrix dot product on the second positionally encoded key data and the positionally encoded query data to generate a second product output; and
a concatenation module, configured to concatenate the first product output and the second product output to generate the attention score.

### Claim 13 (depends on 11)

The system of claim 11, wherein the second multiplication module comprises:
a third batch matrix multiplication module, configured to perform a third matrix dot product according to the attention score and the second value data to generate a third product output;
a fourth batch matrix multiplication module, configured to perform a fourth matrix dot product according to the attention score and the first value data to generate a fourth product output; and
a second adder, configured to add the third product output and the fourth product output to generate the attention output.

### Claim 14 (depends on 12)

The system of claim 12, further comprising:
a softmax module, configured to perform a softmax function to the attention score to generate an attention weight to the second multiplication module, wherein the second element-wise dot product operations are performed according to the first value data, the attention weight, and the second value data to generate the attention output.

### Claim 15 (depends on 11)

The system of claim 11, further comprising:
an input layer normalization module, configured to normalize the hidden state as a normalized hidden state by adjusting a square sum of elements of the hidden state equal to a fixed number; and
an output layer normalization module, configured to normalize the updated hidden output as a normalized updated hidden output by adjusting a square sum of elements of the updated hidden output equal to the fixed number.

### Claim 16 (depends on 11)

The system of claim 11, further comprising:
a ring buffer, configured to save the first value data, the second value data, the first positionally encoded key data, and the second positionally encoded key data;
wherein a part of memory segments of the ring buffer are sliced to generate a first value cache and/or a first key cache, another part of memory segments of the ring buffer are sliced to generate a second value cache and/or a second key cache, the first value cache is configured to save the first value data, the second value cache is configured to save the second value data, the first key cache is configured to save the first positionally encoded key data, and the second key cache is configured to save the second positionally encoded key data.

### Claim 17 (depends on 16)

The system of claim 16, wherein the part of memory segments from the ring buffer are updated by incrementing an offset value of memory addresses after the first value cache and/or a first key cache is saved, after the part of memory segments from the ring buffer are updated, the part of memory segments comprises the first value cache and/or a first key cache, and a part of second positionally encoded key data and/or a part of second value data, and the second value data and/or the second positionally encoded key data are updated after the part of memory segments are shifted.

### Claim 18 (depends on 17)

The system of claim 17, wherein the part of memory segments of the ring buffer are copied according to their initialized memory addresses after the part of memory segments are shifted to hit an end memory segment of the ring buffer.

### Claim 19 (depends on 11)

The system of claim 11, wherein a signal shape format of the first value data and the first positionally encoded key data is represented as (B, N, T, D), a signal shape format of the second value data and the second positionally encoded key data is represented as (B, N, C, D), B is a batch size, T is a token amount, N is an attention head amount, D is a head dimension for each attention head, and C is a used-defined number previously determined.

### Claim 20 (depends on 14)

The system of claim 14, further comprising:
a mask value generator and a third adder, configured to replace null values of the attention score with mask values generated by the mask value generator to generate an updated attention score.
