System and method for balancing compression and read performance in a storage system

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of balancing data compression and read performance of data chunks of a storage system, the operations comprising:

• identifying similar data chunks based on sketches of a plurality of data chunks stored in the storage system;
• scanning a metadata to retrieve chunk identifiers (IDs) and sketches of the plurality of data chunks,
• storing a plurality of entries of a data structure, wherein each of the entries corresponds to one of the sketches and its respective chunk ID, including
• determining that a first sketch of the sketches includes a first feature and a second feature,
• sorting the entries of the data structure based on the first feature,
• identifying a subset of the entries of the data structure that are associated with the first feature, and
• sorting the subset of the entries of the data structure based on the second feature;
• associating a first portion of the similar data chunks as a first group with a first storage container;
• associating with the first storage container one or more data chunks that are dissimilar to the first group but are likely accessed together;
• compressing the first group of the similar data chunks and its associated dissimilar data chunks in a first compression region of the first storage container;
• and
• storing a plurality of storage containers in a persistent storage device of the storage system including the first storage container.

A data processing system, comprising:

• a processor;
• and
• a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
• identifying similar data chunks based on sketches of a plurality of data chunks stored in the storage system;
• scanning a metadata to retrieve chunk identifiers (IDs) and sketches of the plurality of data chunks,
• storing a plurality of entries of a data structure, wherein each of the entries corresponds to one of the sketches and its respective chunk ID, including
• determining that a first sketch of the sketches includes a first feature and a second feature,
• sorting the entries of the data structure based on the first feature,
• identifying a subset of the entries of the data structure that are associated with the first feature, and
• sorting the subset of the entries of the data structure based on the second feature;
• associating a first portion of the similar data chunks as a first group with a first storage container;
• associating with the first storage container one or more data chunks that are dissimilar to the first group but are likely accessed together;
• compressing the first group of the similar data chunks and its associated dissimilar data chunks in a first compression region of the first storage container;
• and
• storing a plurality of storage containers in a persistent storage device of the storage system including the first storage container.