Compiler to determine bounds of an attention mask for tile-level computation skipping
AssigneeNVIDIA Corporation
InventorsMihir Bipin Shah, Rajnish Aggarwal
Life of this patent
Estimating the term…
Abstract
In various examples, apparatuses, systems, processors, methods, and/or machine-readable mediums are used to generate bounding functions for a mask predicate, and to generate instructions to skip computation of values corresponding to one or more portions of an attention matrix masked by the mask predicate. In at least one embodiment, a compiler is used to analyze arbitrary and/or user-defined masks, and to generate bounding functions that may be used by an executing kernel to identify which tiles of an attention matrix can be skipped at runtime.
Claims20
One or more processors comprising processing circuitry to:
- implement a compiler to generate instructions that, when executed by at least one processor, cause the at least one processor to skip computation of one or more masked values in one or more masked portions of at least one data structure representing one or more attention matrices that are masked by a mask predicate, the compiler generating the instructions by at least:
- deriving a plurality of elements based at least on the mask predicate;
- determining pairs of upper and lower bounding expressions corresponding to the plurality of elements;
- generating left and right bounding functions using the pairs of upper and lower bounding expressions corresponding to the plurality of elements;
- and
- using the left and right bounding functions to generate the instructions.
A system comprising one or more processors to:
- generate one or more bounding functions based at least on one or more mask predicates;
- and
- use the one or more bounding functions to generate instructions to cause at least one processor to skip computation of one or more first values corresponding to one or more first portions of at least one data structure representing one or more attention matrices.
A method comprising:
- generating, by a compiler implemented by one or more processors, one or more bounding functions based at least on a mask predicate;
- using, by the compiler, the one or more bounding functions to identify one or more masked tiles and one or more unmasked tiles of a plurality of tiles within a data structure;
- and
- generating, by the compiler, instructions to cause at least one processor to perform at least one computation with respect to the one or more unmasked tiles, and to skip computation with respect to the one or more masked tiles.
Dependencies are inferred from the claim text.