Efficient processing of transformer based models

A non-transitory computer-readable storage medium comprising stored computer executable instructions, the instructions which when executed by a compiler operating on at least one computer processor to:

• execute a transformer having an embedding layer and an encoder layer with an associated self-attention block;
• and wherein
• the at least one computer processor is a Tensor Streaming Processor (TSP) having a Matrix Multiplication module (MXM) and Vector Calculation module (VXM), with the TSP arranged to deterministically process information arranged by the embedding layer and the encoder layer with the associated self-attention mechanism, the information being further modified according to the transformer using a general matrix multiply (GEMM) mapped directly on the MXM and associated accumulator, and wherein at least some set of information is processed to parallelize the execution of GEMMs across all MXM planes, and
• wherein the instructions can be compiled into a binary for execution at the one or more processors, the binary indicating the schedule of execution of the plurality of instructions.

A system, comprising:

• a compiler configured to determine a schedule of execution of the plurality of instructions for execution by the one or more processors that can execute a transformer having an embedding layer and an encoder layer with an associated self-attention block;
• and wherein at least one computer processor is a Tensor Streaming Processor (TSP) having a Matrix Multiplication module (MXM) and Vector Calculation module (VXM), with the TSP arranged to deterministically process information arranged by the embedding layer and the encoder layer with the associated self-attention mechanism, the information being further modified according to the transformer using a general matrix multiply (GEMM) mapped directly on MXM and associated accumulator, and wherein at least some set of information is processed to parallelize the execution of GEMMs across all MXM planes, and wherein
• the compiler can compile the plurality of instructions into a binary, the binary indicating the schedule of execution of the plurality of instructions;
• and allow one or more processors to be configured to execute the binary.

A method, comprising:

• providing a compiler configured to determine a schedule of execution of the plurality of instructions for execution by the one or more processors that can execute a transformer having an embedding layer and an encoder layer with an associated self-attention block;
• and wherein at least one computer processor is a Tensor Streaming Processor (TSP) having a Matrix Multiplication module (MXM) and Vector Calculation module (VXM), with the TSP arranged to deterministically process information arranged by the embedding layer and the encoder layer with the associated self-attention mechanism, the information being further modified according to the transformer using a general matrix multiply (GEMM) mapped directly on MXM and associated accumulator, and wherein at least some set of information is processed to parallelize the execution of GEMMs across all MXM planes, and wherein
• compiling the plurality of instructions into a binary, the binary indicating the schedule of execution of the plurality of instructions;
• and
• allowing one or more processors to be configured to execute the binary.