US 12,001,510 B2 · Method and apparatus for length-aware local tiling in a sparse attention module in a transformer

Abstract

A method and an apparatus for length-aware local tiling in a sparse attention module in a transformer in heterogeneous devices are provided. The method includes that a heterogeneous device including one or more GPUs: divides a transformed sparsity mask into a plurality of first tiles and obtaining one or more effective first tiles from the plurality of first tiles, where each effective first tile includes at least one non-zero element; loads the one or more effective first tiles into a shared memory in the one or more GPUs and loads a plurality of elements in a first matrix corresponding to the one or more effective first tiles into the shared memory; and performs multiplication by a first sampled dense-dense matrix multiplication (SDDMM) kernel in the sparse attention module in the transformer by fetching the one or more effective first tiles and the plurality of elements from the shared memory.

Claims23

Dependent claims

A method for length-aware local tiling in a sparse attention module in a transformer in heterogeneous devices, comprising:

dividing, by a heterogeneous device comprising one or more graphic processing units (GPUs), a transformed sparsity mask into a plurality of first tiles and obtaining one or more effective first tiles from the plurality of first tiles, wherein each effective first tile comprises at least one non-zero element;
loading, by the heterogeneous device, the one or more effective first tiles into a shared memory in the one or more GPUs and loading a plurality of elements in a first matrix corresponding to the one or more effective first tiles into the shared memory;
and
performing, by the heterogeneous device, multiplication by a first sampled dense-dense matrix multiplication (SDDMM) kernel in the sparse attention module in the transformer by fetching the one or more effective first tiles and the plurality of elements from the shared memory.

1 more element

An apparatus for length-aware local tiling in a sparse attention module in a transformer in heterogeneous devices, comprising:

one or more processors;
and
a memory configured to store instructions executable by the one or more processors,
wherein the one or more processors, upon execution of the instructions, are configured to:
divide a transformed sparsity mask into a plurality of first tiles and obtain one or more effective first tiles from the plurality of first tiles, wherein each effective first tile comprises at least one non-zero element;
load the one or more effective first tiles into a shared memory in the one or more processors and load a plurality of elements in a first matrix corresponding to the one or more effective first tiles into the shared memory;
and
perform multiplication by a first sampled dense-dense matrix multiplication (SDDMM) kernel in the sparse attention module in the transformer by fetching the one or more effective first tiles and the plurality of elements from the shared memory.

5 more elements

Dependencies are inferred from the claim text.

Method and apparatus for length-aware local tiling in a sparse attention module in a transformer

Life of this patent

Method and apparatus for length-aware local tiling in a sparse attention module in a transformer

Life of this patent

Abstract

Claims23