Parameter-free attention
AssigneeQUALCOMM Incorporated
InventorsHanno ACKERMANN, Leyla MIRVAKHABOVA, Hong CAI, Fatih Murat PORIKLI, Farhad GHAZVINIAN ZANJANI, Markus NAGEL
Life of this patent
Estimating the term…
Abstract
A processor-implemented method for providing parameter-free attention operations includes receiving, by an attention mechanism of a machine learning model, an input. The attention mechanism generates a set of matrices based on the input. An attention matrix is generated based on a reconstruction objective computed based on a linear combination of the input. The machine learning model computes an output based on the attention matrix.
Claims18
An apparatus, comprising:
- at least one memory;
- and
- at least one processor coupled to the at least one memory, the at least one processor configured to:
- receive, by an attention mechanism of a machine learning model, an input;
- generate, by the attention mechanism, a set of matrices based on the input;
- generate an attention matrix based on a reconstruction objective computed based on a linear combination of the input;
- and
- compute, by the machine learning model, an output based on the attention matrix.
A processor-implemented method performed by at least one processor, the processor-implemented method comprising:
- receiving, by an attention mechanism of a machine learning model, an input;
- generating, by the attention mechanism, a set of matrices based on the input;
- generating an attention matrix based on a reconstruction objective computed based on a linear combination of the input;
- and
- computing, by the machine learning model, an output based on the attention matrix.
An apparatus comprising:
- means for receiving, by an attention mechanism of a machine learning model, an input;
- means for generating, by the attention mechanism, a set of matrices based on the input;
- means for generating an attention matrix based on a reconstruction objective computed based on a linear combination of the input;
- and
- means for computing, by the machine learning model, an output based on the attention matrix.
Dependencies are inferred from the claim text.