Optimized data stream compression using data-dependent chunking

A computing device comprising:

• a memory;
• one or more processors operatively coupled to the memory;
• a compression module to implement compression of a data stream communicated from the computing device, in the memory, wherein the compression module is configured to partition data in the data stream into one or more data chunks based at least in part on previously occurring byte code patterns present within previously occurring data streams different from the data stream;
• a signature module to associate each of the one or more data chunks with a signature in the memory, and to compare each signature of the one or more data chunks with stored signatures to determine a match based at least in part on listing one or more stored data chunks with potentially matching stored signatures;
• wherein the compression module is further configured to:
• perform a forward and backward scan of byte code sequences of matched data chunks to determine a corresponding match-length, wherein a match length is a larger byte code sequence of the corresponding matched data chunk;
• and
• generate an output that includes an element selected from the group consisting of:
• match-length, copy-offset tuples, and byte code sequences remaining without match-lengths;
• and
• a secondary compression module to further compress shorter length matches of the one or more data chunks overlooked by the compression module.

A method of communicating data between a server and one or more client devices comprising:

• partitioning, at the server, a data stream including a byte code sequence into one or more data chunks, the partitioning based at least in part on previously occurring byte code patterns present within previously occurring data streams different from the data stream;
• associating a signature to each of the one or more data chunks;
• comparing each signature of the one or more data chunks with stored signatures to determine a match;
• listing one or more stored data chunks with potentially matching stored signatures;
• performing a forward and backward scan of byte code sequences of matched data chunks to determine a corresponding match-length, wherein a match length is a larger byte code sequence of the corresponding matched data chunk;
• generating an output comprising the match-lengths and a portion of the byte code sequence without matching stored signatures;
• passing the output for further compression, the further compression determining shorter match-lengths overlooked from the partitioning and the performing the forward and backward scan;
• and
• communicating the output to the one or more client devices.