US 2026/0016952 A1 · KV CACHE BLOCK-QUANTIZATION ORIENTED DATA HANDLING

Abstract

A computing system for performing runtime data handling optimization for generative models is provided. The computing system comprises at least one processor and memory comprising a first memory and a second memory, wherein the memory stores instructions that, when executed by the at least one processor, cause the at least one processor to execute a generative model. The computing system computes a first value matrix entry based upon the processing of an input to the generative model. The first value matrix entry is stored in a first memory wherein a first group of value matrix entries is identified. The computing system executes data quantization on the first group of value matrix entries which results in a first quantized value matrix. The first quantized value matrix is added to a second memory where it can be used during generation of the generative model. When the first group of value matrix entries is less than a group size parameter, data padding matrix values are generated and used during execution of the data quantization.

Claims20

Dependent claims

A computing system, comprising:

at least one processor;
and
memory comprising at least a first memory and a second memory, wherein the memory stores instructions that, when executed by the at least one processor, cause the at least one processor to execute a generative model and perform acts comprising:
computing a first value matrix entry;
storing the first value matrix entry in a first memory, wherein the first memory stores value matrix entries computed during execution of the generative model;
identifying a first group of value matrix entries stored in the first memory according to a group size parameter, wherein the value matrix entries stored in the first memory are greater than or equal to the group size parameter;
executing data quantization on the first group of value matrix entries, wherein the data quantization produces a result comprising a first quantized value matrix comprising a plurality of quantization block values;
storing the first quantized value matrix in a second memory;
and
retrieving the first quantized value matrix from the second memory during execution of the generative model.

7 more elements

A method, the method comprising:

computing a first value matrix entry;
storing the first value matrix entry in a first memory, wherein the first memory stores value matrix entries computed during execution of a generative model;
identifying a first group of value matrix entries stored in the first memory according to a group size parameter, wherein the value matrix entries stored in the first memory are greater than or equal to the group size parameter;
executing data quantization on the first group of value matrix entries, wherein the data quantization produces a result comprising a first quantized value matrix comprising a plurality of quantization block values;
storing the first quantized value matrix in a second memory;
and
retrieving the first quantized value matrix from the second memory during execution of the generative model.

4 more elements

A computer-readable storage medium comprising instructions that, when executed by at least one processor of a computing system, cause the at least one processor to perform acts comprising:

computing a first value matrix entry;
storing the first value matrix entry in a first memory, wherein the first memory stores value matrix entries computed during execution of the generative model;
identifying a first group of value matrix entries stored in the first memory according to a group size parameter, wherein the value matrix entries stored in the first memory are greater than or equal to the group size parameter;
executing data quantization on the first group of value matrix entries, wherein the data quantization produces a result comprising a first quantized value matrix comprising a plurality of quantization block values;
storing the first quantized value matrix in a second memory;
and
retrieving the first quantized value matrix from the second memory during execution of the generative model.

4 more elements

Dependencies are inferred from the claim text.

Kv cache block-quantization oriented data handling

Life of this patent

Kv cache block-quantization oriented data handling

Life of this patent

Abstract

Claims20