US 2026/0093988 A1Published application

Attention mechanism adjustment method based on attention score and computing device using the same

AssigneeIndustrial Technology Research Institute

InventorsYao-Hua Chen, Po-Hung Lin, Chih-Tsun Huang

Life of this patent

Estimating the term…

Loading patent…

Abstract

An attention mechanism adjustment method based on attention scores, applicable to Transformer models, is provided. The method includes: for the current Transformer block of the Transformer model, obtaining query matrix, key matrix, and value matrix based on the input sequence; using the self-attention module to generate multiple attention score matrices corresponding to multiple attention heads; before executing the softmax function, performing cross-head column-wise aggregation operation on the attention score matrices to obtain a token importance vector; comparing importance scores with the trained importance score threshold to determine if pruning is needed; executing pruning operations on target tokens that need pruning to obtain pruned attention score matrices; performing softmax function operations on the pruned attention score matrices to obtain a pruned attention probability matrix, where the probability values of the pruned tokens are zero.

Claims26

Dependent claims

An attention mechanism adjustment method based on attention scores, adapted for a transformer model, the method comprising:

for a current transformer block of the transformer model:
obtaining a query matrix, a key matrix and a value matrix corresponding to a received input sequence, wherein the input sequence comprises a plurality of tokens;
generating a plurality of attention score matrices corresponding to the input sequence based on the query matrix and the key matrix, wherein the attention score matrices respectively correspond to a plurality of attention heads;
before executing a softmax function operation, performing a cross-head column-wise aggregation operation on the attention score matrices to obtain a token importance vector corresponding to the input sequence, wherein a plurality of elements of the token importance vector respectively represent a plurality of importance scores corresponding to the tokens;
determining whether each token of the input sequence needs to be pruned through the importance score of each element in the token importance vector and a trained importance score threshold;
in response to determining that one or more target tokens need to be pruned, changing the attention score matrices into a plurality of pruned attention score matrices by performing a pruning operation on the one or more target tokens;
and
performing the softmax function operation on the pruned attention score matrices to obtain a pruned attention probability matrix, wherein one or more probability values corresponding to the pruned one or more target tokens in the pruned attention probability matrix are zero, so as to optimize the attention mechanism of the transformer model, reduce invalid operations in subsequent calculations, and thereby improve calculation efficiency and inference speed of the transformer model.

5 more elements

A computing device, adapted for executing a transformer model that adjusts attention mechanism based on attention scores, the computing device comprising:

a processor;
a memory, coupled to the processor;
and
a storage device, coupled to the processor, the storage device storing a plurality of program code modules, wherein the processor is configured to execute the program code modules to:
for a current transformer block of the transformer model:
obtain, via a query-key-value (QKV) generation module, a query matrix, a key matrix and a value matrix corresponding to a received input sequence, wherein the input sequence comprises a plurality of tokens;
generate, via an attention calculation module, a plurality of attention score matrices corresponding to the input sequence based on the query matrix and the key matrix, wherein the attention score matrices respectively correspond to a plurality of attention heads;
before executing a softmax function operation, perform, via a pruning module, a cross-head column-wise aggregation operation on the attention score matrices to obtain a token importance vector corresponding to the input sequence, wherein a plurality of elements of the token importance vector respectively represent a plurality of importance scores corresponding to the tokens;
determine, via the pruning module, whether each token of the input sequence needs to be pruned through the importance score of each element in the token importance vector and a trained importance score threshold;
in response to determining that one or more target tokens need to be pruned, change, via the pruning module, the attention score matrices into a plurality of pruned attention score matrices by performing a pruning operation on the one or more target tokens;
and
perform the softmax function operation on the pruned attention score matrices to obtain a pruned attention probability matrix, wherein one or more probability values corresponding to the pruned one or more target tokens in the pruned attention probability matrix are zero, so as to optimize the attention mechanism of the transformer model, reduce invalid operations in subsequent calculations, and thereby improve calculation efficiency and inference speed of the transformer model.

9 more elements

Dependencies are inferred from the claim text.