Selection of hash key sizes for data deduplication

A system comprising:

• a processor;
• and
• a memory comprising code stored thereon that, when executed, performs a method of processing data comprising:
• receiving, using a processor, a data chunk;
• determining, using a processor, a metric value denoting a degree of compressibility of the data chunk;
• selecting, using a processor and in accordance with the metric value denoting the compressibility of the data chunk, a first size of a plurality of sizes, wherein the plurality of sizes denote different sizes of hash keys and hash values used in connection with data deduplication processing of data chunks, wherein the first size denotes a hash key size and a hash value size selected for use in connection with the data deduplication processing of the data chunk;
• and
• performing, using a processor, the data deduplication processing of the data chunk, wherein the data deduplication processing of the data chunk includes using a first hash value of the first size for the data chunk to determine whether the data chunk is a duplicate of a first existing data chunk of a hash table, wherein the first hash value is stored in a storage location of the first size, the hash table includes hash keys of the first size, and wherein the first existing data chunk is associated with a first hash key of the first size, and wherein the data deduplication processing for the data chunk further comprises:
• mapping the first hash value to the first hash key of the hash table matching the first hash value, wherein the first hash key is used an index into the hash table and wherein the first hash key is a value stored in a storage location of the first size;
• and
• determining whether the first existing data chunk of the hash table matches the data chunk.

A non-transitory computer readable medium comprising code stored thereon that, when executed, performs a method of processing data comprising:

• receiving, using a processor, a data chunk;
• determining, using a processor, a metric value denoting a degree of compressibility of the data chunk;
• selecting, using a processor and in accordance with the metric value denoting the compressibility of the data chunk, a first size of a plurality of sizes, wherein the plurality of sizes denote different sizes of hash keys and hash values used in connection with data deduplication processing of data chunks, wherein the first size denotes a hash key size and a hash value size selected for use in connection with the data deduplication processing of the data chunk;
• and
• performing, using a processor, the data deduplication processing of the data chunk, wherein the data deduplication processing of the data chunk includes using a first hash value of the first size for the data chunk to determine whether the data chunk is a duplicate of a first existing data chunk of a hash table, wherein the first hash value is stored in a storage location of the first size, the hash table includes hash keys of the first size, and wherein the first existing data chunk is associated with a first hash key of the first size, and wherein the data deduplication processing for the data chunk further comprises:
• mapping the first hash value to the first hash key of the hash table matching the first hash value, wherein the first hash key is used an index into the hash table and wherein the first hash key is a value stored in a storage location of the first size;
• and
• determining whether the first existing data chunk of the hash table matches the data chunk.