# US 2025/0094775 A1 · Systems and methods for static cached decoding

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094775 A1
- Document ID: US20250094775A1
- Status: Published application
- Kind: A1
- Application number: 18/468,574
- Filed: 2023-09-15
- Published: 2025-03-20
- Earliest claimed: 2023-09-15
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Shaojie ZHUO; Ramchalam KINATTINKARA RAMAKRISHNAN; Xiaopeng ZHANG; Yicheng LIN; Chenzheng SU; Liang SHEN
- Web page: https://patentgrep.com/patent/US20250094775A1

## CPC

- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0475: Generative networks
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Cached decoding systems and techniques are described. A system (e.g., decoder) receives an input token (e.g., input vector). The system applies a projection tensor (e.g., a projection matrix) to the input token to generate a feature tensor (e.g., a key tensor or a value tensor). The system processes at least the feature tensor and at least one previous feature tensor using at least one attention calculation to generate an output token. The at least one previous feature tensor is retrieved from a buffer. The at least one previous feature tensor can be stored in the buffer after having been previously calculated based on application of the projection tensor to a previous input token (e.g., from a previous iteration before the iteration in which the input token is received).

## Claims

### Claim 1

An apparatus for cached decoding, the apparatus comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
receive an input token;
apply a projection tensor to the input token to generate a feature tensor; and
process at least the feature tensor and at least one previous feature tensor using at least one attention calculation to generate an output token, the at least one previous feature tensor retrieved from a buffer, and the at least one previous feature tensor previously calculated based on application of the projection tensor to a previous input token.

### Claim 2 (depends on 1)

The apparatus of claim 1,
wherein the feature tensor is a key feature tensor,
wherein the projection tensor is a key projection tensor, and
wherein the buffer is a key buffer.

### Claim 3 (depends on 1)

The apparatus of claim 1,
wherein the feature tensor is a value feature tensor,
wherein the projection tensor is a value projection tensor, and
wherein the buffer is a value buffer.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is configured to:
output an output that is based on at least the output token.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is configured to:
store the feature tensor in the buffer.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the at least one processor is configured to:
overwrite an invalid portion of the buffer with the feature tensor to store the feature tensor in the buffer.

### Claim 7 (depends on 5)

The apparatus of claim 5, wherein the at least one processor is configured to:
discard an oldest feature tensor from the buffer before storing the feature tensor in the buffer, wherein the oldest feature tensor is an oldest one of a plurality of feature tensors stored in the buffer, the plurality of feature tensors including the at least one previous feature tensor.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is configured to:
receive a second input token;
apply the projection tensor to the second input token to generate a second feature tensor; and
process at least the second feature tensor, the feature tensor, and the at least one previous feature tensor using the at least one attention calculation to generate a second output token, the feature tensor and the at least one previous feature tensor retrieved from the buffer.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the at least one processor is configured to:
output an output that is based on at least the output token and the second output token.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is configured to:
retrieve the at least one previous feature tensor from the buffer.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein the input token is based on an output of an encoder, wherein the at least one attention calculation and the projection tensor are part of a decoder.

### Claim 12 (depends on 11)

The apparatus of claim 11, further comprising:
the decoder, wherein an output of the decoder is based on the output token.

### Claim 13 (depends on 11)

The apparatus of claim 11,
wherein an input of the encoder includes a first string of text,
wherein an output of the decoder includes a second string of text that is based on the first string of text.

### Claim 14 (depends on 1)

The apparatus of claim 1,
wherein the at least one attention calculation receives three inputs, the three inputs including a query input and a key input and a value input, and
wherein one of the three inputs includes at least the feature tensor and the at least one previous feature tensor.

### Claim 15 (depends on 1)

The apparatus of claim 1, wherein the at least one attention calculation includes a scaling function that uses a scaling factor dk.

### Claim 16 (depends on 1)

The apparatus of claim 1, wherein the at least one attention calculation includes a mask configured to confine an attention span.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein the mask is dependent on an iteration of the at least one attention calculation.

### Claim 18 (depends on 1)

The apparatus of claim 1, wherein the at least one attention calculation includes a softmax function configured to normalize at least one weight value.

### Claim 19 (depends on 1)

The apparatus of claim 1, wherein an inference graph associated with the at least one attention calculation is static.

### Claim 20 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is configured to:
initialize the buffer, wherein the buffer is sized according to a first dimension and a second dimension, wherein the first dimension of the buffer is based on a maximum context length, wherein the second dimension of the buffer is based on a size of the input token.

### Claim 21 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is configured to:
maintain a counter tracking a number of feature tensors cached in the buffer.

### Claim 22 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is configured to:
maintain a counter tracking a number of iterations of the at least one attention calculation.

### Claim 23

A method for cached decoding, the method comprising:
receiving an input token;
applying a projection tensor to the input token to generate a feature tensor; and
processing at least the feature tensor and at least one previous feature tensor using at least one attention calculation to generate an output token, the at least one previous feature tensor retrieved from a buffer, and the at least one previous feature tensor previously calculated based on application of the projection tensor to a previous input token.

### Claim 24 (depends on 23)

The method of claim 23, wherein the feature tensor is a key feature tensor, wherein the projection tensor is a key projection tensor, and wherein the buffer is a key buffer.

### Claim 25 (depends on 23)

The method of claim 23, wherein the feature tensor is a value feature tensor, wherein the projection tensor is a value projection tensor, and wherein the buffer is a value buffer.

### Claim 26 (depends on 23)

The method of claim 23, further comprising:
outputting an output that is based on at least the output token.

### Claim 27 (depends on 23)

The method of claim 23, further comprising:
storing the feature tensor in the buffer.

### Claim 28 (depends on 27)

The method of claim 27, further comprising:
overwriting an invalid portion of the buffer with the feature tensor to store the feature tensor in the buffer.

### Claim 29 (depends on 27)

The method of claim 27, further comprising:
discarding an oldest feature tensor from the buffer before storing the feature tensor in the buffer, wherein the oldest feature tensor is an oldest one of a plurality of feature tensors stored in the buffer, the plurality of feature tensors including the at least one previous feature tensor.

### Claim 30 (depends on 27)

The method of claim 27, further comprising:
retrieving the at least one previous feature tensor from the buffer.
