System and method for improving data compression

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor perform operations for improving compression of data, the operations comprising:

• partitioning by a segmenter a first sequence of data into a plurality of data chunks in a first sequence order according to a predetermined chunking algorithm;
• generating by the segmenter a chunk representation for each of the data chunks by hashing a portion of a corresponding data chunk using a predetermined hash function;
• determining by a similarity detector similarity of the data chunks by comparing hash values of the data chunks;
• reorganizing by a reorganizer the data chunks into a second sequence order based on the similarity of the data chunks, including grouping similar data chunks within the second sequence order that have the same or similar chunk representations, the second sequence order being different from the first sequence order;
• compressing by a compressor the data chunks in the second sequence order into a second sequence of data, wherein the compressed data chunks result in a smaller storage space required to store the compressed data chunks;
• and
• storing in a persistent storage device a mapping recipe mapping the first sequence order to the second sequence order, wherein the mapping recipe is utilized to restore the data chunks subsequently.

A data processing system, comprising:

• a processor;
• and
• a memory coupled to the processor for storing instructions, which when executed by the processor, cause the processor to perform operations for improving compression of data, the operations including
• partitioning a segmenter a first sequence of data into a plurality of data chunks in a first sequence order according to a predetermined chunking algorithm,
• generating by the segmenter a chunk representation for each of the data chunks by hashing a portion of a corresponding data chunk using a predetermined hash function,
• determining by a similarity detector similarity of the data chunks by comparing hash values of the data chunks,
• reorganizing by a reorganizer the data chunks into a second sequence order based on the similarity of the data chunks, including grouping similar data chunks within the second sequence order that have the same or similar chunk representations, the second sequence order being different from the first sequence order,
• compressing by a compressor the data chunks in the second sequence order into a second sequence of data, wherein the compressed data chunks result in a smaller storage space required to store the compressed data chunks, and
• store in a persistent storage device a mapping recipe mapping the first sequence order to the second sequence order, wherein the mapping recipe is utilized to restore the data chunks subsequently.