Heterogeneous compression architecture for optimized compression ratio

A method comprising:

• identifying, by an accelerator circuit, a byte in a byte stream;
• determining whether a first byte string starting from a first byte position of the byte matches a second byte string starting from a second byte position;
• responsive to determining that the first byte string matches the second byte string, generating a token comprising a first symbol encoding a length of the first byte string and a second symbol encoding a byte distance between the first byte position and the second byte position;
• and
• responsive to determining that the first byte string does not match another byte string, generating the token comprising the first symbol comprising the byte and a second symbol encoding a determined value.

A processing device to:

• receive a token stream comprising a plurality of tokens generated based on a byte stream comprising a plurality of bytes, wherein each token in the token stream comprises at least one symbol associated with a respective byte in the byte stream, and wherein the at least one symbol represents one of the respective byte, a length of a first byte string starting from the respective byte, or a byte distance between the first byte string and a matching second byte string;
• generate a graph comprising a plurality of nodes and edges based on the token stream, wherein each token in the token stream is associated with a respective node connected by at least one edge to another node, and wherein the at least one edge is associated with a cost function to encode the at least one symbol stored in the each token;
• identify, based on the graph, a path between a first node associated with a beginning token of the token stream and an end node associated with a last token of the token stream, wherein the path comprises a subset of nodes and edges linking the subset of nodes;
• and
• perform variable-length encoding of a subset of tokens associated with the subset of nodes to generate an output data.