Lossless compression of llm parameters with grouped huffman encoding

A method comprising:

• packing parameters of a neural network model into parameter groups;
• performing block-based Huffman encoding on the parameter groups to generate encoded parameter blocks;
• transferring the encoded parameter blocks and a decode table to an accelerator device;
• performing parallel Huffman decoding on the encoded parameter blocks via the decode table to recover the parameter groups;
• unpacking the parameter groups to recover parameters of the neural network model;
• and
• performing an inference operation to generate an inference result.

A data processing system comprising:

• a memory device;
• and
• an accelerator device coupled with the memory device, the accelerator device comprising a host interface and a processing cluster coupled with the host interface, the processing cluster including a plurality of processing resources, a processing resource of the plurality of processing resources configured to:
• receive a decode table and plurality of blocks of encoded parameters of a neural network model;
• perform a parallel Huffman decoding operation on the plurality of blocks of encoded parameters based on the decode table to concurrently decode multiple blocks of the plurality of blocks of encoded parameters, the plurality of blocks of encoded parameters decoded into a plurality of parameter groups;
• unpack each of the plurality of parameter groups into a plurality of parameters of the neural network model during an inference operation performed via the neural network model;
• and
• store or output a result of the inference operation.