Tunable data fingerprinting for optimizing data deduplication

A computer program product of tunable data fingerprinting for optimizing data deduplication performance of a storage system, said program product comprising a non-transitory computer readable storage medium having computer usable program code embodied therewith, said program code being executable by a storage system to:

• receive a file to be stored by the storage system;
• determine data fingerprinting parameters for turning a rolling-hash algorithm for chunking data in the file, said parameters are based on at least one of:
• file type usage in the storage system;
• a type of content in the file;
• and the file's size, and said parameters include at least one of:
• a minimum chunk size;
• average expected chunk size;
• maximum chunk size;
• and chunking window size;
• tune the rolling-hash algorithm based on said parameters;
• chunk data in the file using said tuned algorithm;
• generate a content identifier for each of the file's data chunks by applying a collision-resistant hashing algorithm to each of the chunks of data;
• and
• process the chunks of data that are unique, the content identifier for each of the chunks of data that are unique, and references to the chunks of data that are unique, said process includes:
• when an index of the storage system includes the content identifier, then:
• identify the chunk of data corresponding to the content identifier as being a duplicated chunk of data;
• removing the duplicated chunk of data from the system;
• and add a reference to the identified chunk of data, and when an index of the system does not include the content identifier, then:
• identify the chunk of data corresponding to the content identifier as being a unique chunk of data;
• identify the content identifier as a unique content identifier;
• adding the unique content identifier to the index;
• storing the identified chunk of data in the storage system;
• and storing said parameters for subsequent verification of parameters used to generate a content identifier.

A system of tunable data fingerprinting for optimizing data deduplication performance of a storage system, comprising:

• a data store for storing data in the storage system;
• memory for caching data stored or to be stored in the storage system;
• a processor for executing computer usable program code to:
• receive a file to be stored by the storage system;
• determine data fingerprinting parameters for turning a rolling-hash algorithm for chunking data in the file, said parameters are based on at least one of:
• file type usage in the storage system;
• a type of content in the file;
• and the file's size, and said parameters include at least one of:
• a minimum chunk size;
• average expected chunk size;
• maximum chunk size;
• and chunking window size;
• tune the rolling-hash algorithm based on said parameters;
• chunk data in the file using said tuned algorithm;
• generate a content identifier for each of the file's data chunks by applying a collision-resistant hashing algorithm to each of the chunks of data;
• and
• process the chunks of data that are unique, the content identifier for each of the chunks of data that are unique, and references to the chunks of data that are unique, said process includes:
• when an index of the storage system includes the content identifier, then:
• identify the chunk of data corresponding to the content identifier as being a duplicated chunk of data;
• removing the duplicated chunk of data from the system;
• and add a reference to the identified chunk of data, and when an index of the system does not include the content identifier, then:
• identify the chunk of data corresponding to the content identifier as being a unique chunk of data;
• identify the content identifier as a unique content identifier;
• adding the unique content identifier to the index;
• storing the identified chunk of data in the data store;
• and storing said parameters for subsequent verification of parameters used to generate a content identifier.