Cache data compression method for transformer-based neural network model and electronic device for performing the same

A non-transitory computer-readable storage medium storing instructions that, when executed by a processor of an electronic device, cause the electronic device to perform:

• obtaining a query tensor, a key tensor, and a value tensor corresponding to tokens;
• determining token importance for each of the tokens based on the query tensor and the key tensor;
• clustering the key tensor and the value tensor into a plurality of clusters based on the determined token importance;
• determining channel importance of channels included in at least one of the key tensor and the value tensor in each of the plurality of clusters;
• generating compressed key-value data by pruning at least one of the channels of the key tensor and the value tensor based on the determined channel importance;
• and
• storing the compressed key-value data as key-value cache data for a transformer-based neural network model.

An electronic device for performing a cache data compression method for a transformer-based neural network model, the electronic device comprising:

• a memory storing instructions;
• and
• a processor configured to execute the instructions,
• wherein, when the instructions are executed by the processor, the electronic device is configured to:
• obtain a query tensor, a key tensor, and a value tensor corresponding to tokens;
• determine token importance for each of the tokens based on the query tensor and the key tensor;
• cluster the key tensor and the value tensor for the tokens into a plurality of clusters based on the determined token importance;
• determine channel importance of channels included in at least one of the key tensor and the value tensor in each of the plurality of clusters;
• generate compressed key-value data by pruning at least one of the channels of the key tensor and the value tensor based on the determined channel importance;
• and
• store the compressed key-value data in the memory as key-value cache data for the transformer-based neural network model.