System and method for balancing compression and read performance in a storage system
AssigneeEMC IP Holding Company LLC
InventorsFrederick Douglis, Philip Shilane, Grant Wallace
Life of this patent
Estimating the term…
Abstract
Techniques for balancing data compression and read performance of data chunks of a storage system are described herein. According to one embodiment, similar data chunks are identified based on sketches of a plurality of data chunks stored in the storage system. A first portion of the similar data chunks as a first group is associated with a first storage area. The first storage area is associated with one or more data chunks that are dissimilar to the first group but are likely accessed together. The first group of the similar data chunks and its associated dissimilar data chunks are compressed and stored in the first storage area.
Claims21
A computer-implemented method for balancing data compression and read performance of data chunks of a storage system, the method comprising:
- identifying similar data chunks based on sketches of a plurality of data chunks stored in the storage system;
- scanning a metadata to retrieve chunk identifiers (IDs) and sketches of the plurality of data chunks,
- storing a plurality of entries of a data structure, wherein each of the entries corresponds to one of the sketches and its respective chunk ID, including
- determining that a first sketch of the sketches includes a first feature and a second feature,
- sorting the entries of the data structure based on the first feature,
- identifying a subset of the entries of the data structure that are associated with the first feature, and
- sorting the subset of the entries of the data structure based on the second feature, wherein the second feature is determined based on a plurality of features of a data chunk that is associated with the chunk ID;
- associating a first portion of the similar data chunks as a first group with a first storage container;
- associating with the first storage container one or more data chunks that are dissimilar to the first group but are likely accessed together;
- compressing the first group of the similar data chunks and its associated dissimilar data chunks in a first compression region of the first storage container;
- and
- storing a plurality of storage containers in a persistent storage device of the storage system including the first storage container.
A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of balancing data compression and read performance of data chunks of a storage system, the operations comprising:
- identifying similar data chunks based on sketches of a plurality of data chunks stored in the storage system;
- scanning a metadata to retrieve chunk identifiers (IDs) and sketches of the plurality of data chunks,
- storing a plurality of entries of a data structure, wherein each of the entries corresponds to one of the sketches and its respective chunk ID, including
- determining that a first sketch of the sketches includes a first feature and a second feature,
- sorting the entries of the data structure based on the first feature,
- identifying a subset of the entries of the data structure that are associated with the first feature, and
- sorting the subset of the entries of the data structure based on the second feature, wherein the second feature is determined based on a plurality of features of a data chunk that is associated with the chunk ID;
- associating a first portion of the similar data chunks as a first group with a first storage container;
- associating with the first storage container one or more data chunks that are dissimilar to the first group but are likely accessed together;
- compressing the first group of the similar data chunks and its associated dissimilar data chunks in a first compression region of the first storage container;
- and
- storing a plurality of storage containers in a persistent storage device of the storage system including the first storage container.
A data processing system, comprising:
- a processor;
- and
- a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
- identifying similar data chunks based on sketches of a plurality of data chunks stored in the storage system;
- scanning a metadata to retrieve chunk identifiers (IDs) and sketches of the plurality of data chunks,
- storing a plurality of entries of a data structure, wherein each of the entries corresponds to one of the sketches and its respective chunk ID, including
- determining that a first sketch of the sketches includes a first feature and a second feature,
- sorting the entries of the data structure based on the first feature,
- identifying a subset of the entries of the data structure that are associated with the first feature, and
- sorting the subset of the entries of the data structure based on the second feature, wherein the second feature is determined based on a plurality of features of a data chunk that is associated with the chunk ID;
- associating a first portion of the similar data chunks as a first group with a first storage container;
- associating with the first storage container one or more data chunks that are dissimilar to the first group but are likely accessed together;
- compressing the first group of the similar data chunks and its associated dissimilar data chunks in a first compression region of the first storage container;
- and
- storing a plurality of storage containers in a persistent storage device of the storage system including the first storage container.
Dependencies are inferred from the claim text.