# US 2026/0093956 A1 · Parameter-free attention

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2026/0093956 A1
- Document ID: US20260093956A1
- Status: Published application
- Kind: A1
- Application number: 19/340,783
- Filed: 2025-09-25
- Published: 2026-04-02
- Earliest claimed: 2024-09-27
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Hanno ACKERMANN; Leyla MIRVAKHABOVA; Hong CAI; Fatih Murat PORIKLI; Farhad GHAZVINIAN ZANJANI; Markus NAGEL
- Web page: https://patentgrep.com/patent/US20260093956A1

## CPC

- G06N 3/045: Combinations of networks
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A processor-implemented method for providing parameter-free attention operations includes receiving, by an attention mechanism of a machine learning model, an input. The attention mechanism generates a set of matrices based on the input. An attention matrix is generated based on a reconstruction objective computed based on a linear combination of the input. The machine learning model computes an output based on the attention matrix.

## Claims

### Claim 1

An apparatus, comprising:
at least one memory; and
at least one processor coupled to the at least one memory, the at least one processor configured to:
receive, by an attention mechanism of a machine learning model, an input;
generate, by the attention mechanism, a set of matrices based on the input;
generate an attention matrix based on a reconstruction objective computed based on a linear combination of the input; and
compute, by the machine learning model, an output based on the attention matrix.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is further configured to compute the reconstruction objective based on a remixing vector that determines a correlation of features between a first matrix in the set of matrices and a second matrix in the set of matrices.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is further configured to compute the attention matrix using a pseudo-inverse of a third matrix of the set of matrices.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the at least one processor is further configured to approximate the pseudo-inverse using an eigenvalue decomposition technique.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is further configured to apply an iterative solver based on l2-norm optimization to generate the attention matrix.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the at least one processor is further configured to apply a parameter that controls an l1-norm optimization to reconstruct the attention matrix.

### Claim 7

A processor-implemented method performed by at least one processor, the processor-implemented method comprising:
receiving, by an attention mechanism of a machine learning model, an input;
generating, by the attention mechanism, a set of matrices based on the input;
generating an attention matrix based on a reconstruction objective computed based on a linear combination of the input; and
computing, by the machine learning model, an output based on the attention matrix.

### Claim 8 (depends on 7)

The processor-implemented method of claim 7, further comprising computing the reconstruction objective based on a remixing vector that determines a correlation of features between a first matrix in the set of matrices and a second matrix in the set of matrices.

### Claim 9 (depends on 7)

The processor-implemented method of claim 7, further comprising computing the attention matrix using a pseudo-inverse of a third matrix of the set of matrices.

### Claim 10 (depends on 9)

The processor-implemented method of claim 9, further comprising approximating the pseudo-inverse using an eigenvalue decomposition technique.

### Claim 11 (depends on 7)

The processor-implemented method of claim 7, further comprising applying an iterative solver based on l2-norm optimization to generate the attention matrix.

### Claim 12 (depends on 11)

The processor-implemented method of claim 11, further comprising applying a parameter that controls an l1-norm optimization to reconstruct the attention matrix.

### Claim 13

An apparatus comprising:
means for receiving, by an attention mechanism of a machine learning model, an input;
means for generating, by the attention mechanism, a set of matrices based on the input;
means for generating an attention matrix based on a reconstruction objective computed based on a linear combination of the input; and
means for computing, by the machine learning model, an output based on the attention matrix.

### Claim 14 (depends on 13)

The apparatus of claim 13, further comprising means for computing the reconstruction objective based on a remixing vector that determines a correlation of features between a first matrix in the set of matrices and a second matrix in the set of matrices.

### Claim 15 (depends on 13)

The apparatus of claim 13, further comprising means for computing the attention matrix using a pseudo-inverse of a third matrix of the set of matrices.

### Claim 16 (depends on 15)

The apparatus of claim 15, further comprising means for approximating the pseudo-inverse using an eigenvalue decomposition technique.

### Claim 17 (depends on 13)

The apparatus of claim 13, further comprising means for applying an iterative solver based on l2-norm optimization to generate the attention matrix.

### Claim 18 (depends on 17)

The apparatus of claim 17, further comprising means for applying a parameter that controls an l1-norm optimization to reconstruct the attention matrix.
