US 12,182,028 B1 · Method and apparatus to cache key-value data in low-precision numerics for efficient generative transformer execution

Abstract

A transformer compute apparatus and method of operation therefor. The apparatus receives matrix inputs in a first format and generates projection tokens from these inputs. Among others, the apparatus includes a first cache device configured for processing first projection tokens and a second cache device configured for processing second projection tokens. The first cache device stores the first projection tokens in a first cache region and stores these tokens converted to a second format in a second cache region. The second cache device stores the second projection tokens converted to the second format in a first cache region and stores the converted second projection tokens after being transposed. Then, a compute device performs various matrix computations with the converted first projection tokens and transposed second projection tokens. Re-processing data and expensive padding and de-padding operations for transposed storage and byte alignment can be avoided using this caching process.

A method of operating a transformer compute apparatus, the method comprising:

receiving, by an input buffer (IB) device coupled a crossbar device, a plurality of matrix inputs in a first format;
wherein a compute device is coupled to the IB device and the crossbar device, an output buffer (OB) device is coupled to the compute device and the crossbar device, a Single Instruction, Multiple Data (SIMD) device is coupled to the OB device, a crossbar converter device is coupled to the OB device and the crossbar device, and a memory device is coupled to the crossbar device;
wherein a cache memory device is coupled to the crossbar device and the memory device, the cache memory device having at least a first cache device and a second cache device;
wherein the first cache device comprises at least a first cache region, a second cache region, and a third cache region;
and
wherein the second cache device comprises at least a first cache region, a second cache region, and a third cache region;
storing, by each of the third cache region of the first cache device and the third cache region of the second cache device, the plurality of matrix inputs;
determining a first projection token, a second projection token, and a third projection token in the first format for each of the plurality of matrix inputs using the compute device;
storing the plurality of first projection tokens, the plurality of second projection tokens, and the plurality of third projection tokens using the OB device;
storing, by the first cache region of the first cache device, the plurality of first projection tokens;
determining, by the crossbar converter device, a plurality of converted second projection tokens in a second format using the plurality of second projection tokens from the OB device;
storing, by the first cache region of the second cache device, the plurality of converted second projection tokens in a blocking configuration;
determining, by the crossbar converter device, a plurality of converted third projection tokens in the second format using the plurality of third projection tokens from the OB device;
storing, by the memory device, the plurality of converted third projection tokens in the blocking configuration;
determining, by the crossbar converter device, a plurality of converted first projection tokens in the second format using the plurality of first projection tokens from the first cache region of the first cache device;
storing, by the second cache region of the first cache device, the plurality of converted first projection tokens in the blocking configuration;
determining, by the crossbar device, a plurality of transposed second projection tokens using the plurality of converted second projection tokens from the first cache region of the second cache device;
storing, by the second cache region of the second cache device, the plurality of transposed second projection tokens in the blocking configuration;
determining, by the compute device and the SIMD device, a plurality of normalized score values using the plurality of converted third projection tokens from the memory device and the plurality of transposed second projection tokens from the second cache region of the second cache device;
storing, by the OB device, the plurality of normalized score values;
storing, by the memory device, the plurality of normalized score values;
determining, by the compute device, a plurality of weighted first projection tokens using the plurality of normalized score values from the memory device and the plurality of converted first projection tokens from the second cache region of the first cache device;
storing, by the OB device, the plurality of weighted first projection tokens;
storing, by the memory device, the plurality of weighted first projection tokens;
determining, by the compute device, a weighted tokens sum using the plurality of weighted first projection tokens from the memory device;
storing, by the OB device, the weighted tokens sum;
and
storing, by the memory device, the weighted tokens sum.

25 more elements

Method and apparatus to cache key-value data in low-precision numerics for efficient generative transformer execution

Life of this patent

Method and apparatus to cache key-value data in low-precision numerics for efficient generative transformer execution

Life of this patent

Abstract

Claims20