System and method for improving data compression of a storage system in an online manner

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for improving data compression of data chunks of a storage system, the operations comprising:

• in response to a sequence of data to be stored, partitioning the sequence of data into a plurality of data chunks according to a predetermined chunking algorithm;
• generating a sketch for each data chunk of the data chunks based on one or more features extracted from the data chunk;
• performing a lookup operation in a sketch index based on the sketch of the each data chunk to determine a compression region identifier (ID) of an existing compression region, wherein the sketch index maps a particular sketch to a particular compression region;
• associating said each data chunk of the data chunks of the sequence of data with one of a plurality of groups based on the particular sketch, wherein each group of the plurality of groups is represented by a sketch, wherein the associating each data chunk of the data chunks of the sequence of data comprises merging the data chunk of the sequence of data with data chunks of an existing compression region identified by a corresponding compression region ID, wherein the merging comprises reorganizing the data chunks of the sequence of data and the data chunks of the existing compression region from an original sequence order to a second sequence order, similar data chunks being positioned adjacent to each other;
• and
• compressing and storing data chunks of the each group in a corresponding existing compression region of the storage system so that the similar data chunks are compressed and stored in the same compression region, wherein the compressing and storing data chunks of each group comprises writing data chunks of each group of the groups whose data chunks reach a predetermined threshold to a respective compression region and reclaiming a previous compression region space.

A data processing system, comprising:

• a processor;
• and
• a memory coupled to the processor for storing instructions, which when executed by the processor, cause the processor to perform operations, the operations including:
• in response to a sequence of data to be stored, partitioning the sequence of data into a plurality of data chunks according to a predetermined chunking algorithm,
• generating a sketch for each data chunk of the data chunks based on one or more features extracted from the data chunk,
• performing a lookup operation in a sketch index based on the sketch of the data chunk to determine a compression region identifier (ID) of an existing compression region, wherein the sketch index maps a particular sketch to a particular compression region,
• associating said each data chunk of the data chunks of the sequence of data with one of a plurality of groups based on the particular ketch, wherein each group is represented by a sketch, wherein the associating each data chunk of the data chunks of the sequence of data comprises merging the data chunk of the sequence of data with data chunks of an existing compression region identified by a corresponding compression region ID, wherein the merging comprises reorganizing the data chunks of the sequence of data and the data chunks of the existing compression region from an original sequence order to a second sequence order, similar data chunks being positioned adjacent to each other, and
• compressing and storing data chunks of each group in a corresponding existing compression region of the storage system so that the similar data chunks are compressed and stored in the same compression region, wherein the compressing and storing data chunks of each group comprises writing data chunks of each group of the groups whose data chunks reach a predetermined threshold to a respective compression region and reclaiming a previous compression region space.