Spatial architecture for attention mechanisms
AssigneeUntether Ai Corporation
InventorsAndrew Vincent ROCK, David RAMON PRADOS, John KITAMURA
Life of this patent
Estimating the term…
Abstract
The present specification discloses a computing device architecture and method for executing computationally intensive attention mechanisms, as commonly utilized in transformer models for artificial intelligence. In an embodiment, the device includes a row of interconnected processing elements (PEs), each linked to dedicated coefficient memory units (CRAMs) and controlled by a central controller. By employing a diagonal-offset and transposition technique for matrix coefficients, the architecture enables efficient execution of Generalized Matrix-Vector (GEMV) operations.
Claims19
A method of operating a computing device having a plurality of processing elements connected to a controller, the method comprising:
- receiving, by the controller, at least a portion of an input matrix into the processing elements;
- the input matrix having undergone a pre-scaling operation;
- loading, by the controller, a coefficient into a memory unit associated with each of the processing elements in a diagonally-offset configuration;
- transposing the coefficients by shifting each coefficient from its respective memory unit in a first processing element to another memory location in an adjacent processing element;
- executing, within the processing elements, a computational operation based on the transposed coefficients within the processing elements;
- and,
- controlling, by the controller, an output device based on the executed computational operation.
Dependencies are inferred from the claim text.