Multi-level inline data deduplication

A server configured to perform multi-level inline data deduplication, the server comprising:

• a memory configured to store instructions;
• and
• a deduplication application configured to be executed by the server in conjunction with the stored instructions, wherein the deduplication application is configured to:
• fingerprint data to be deduplicated to generate fingerprinted content;
• divide, by a chunk manager of the deduplication application, the data into variable-size chunks by an employment of a content dependent chunking (CDC) algorithm based on Rabin fingerprinting to the data, wherein a size of each of the variable-size chunks depends on a size of corresponding fingerprinted content;
• divide, by the chunk manager, each of the variable-size chunks into fixed-size sub-chunks;
• compute the fingerprinted content of the fixed-size sub-chunks, wherein each of the fingerprinted content for the fixed-size sub-chunks is a 20-byte hash;
• store the fingerprinted content of the fixed-size sub-chunks in a two-dimensional array of cells, wherein each cell of the two-dimensional array of cells includes an original fingerprint and an array of a number of bits, and wherein the number of bits reduce a search time to locate the corresponding fingerprinted content for the variable-size chunks by Bloom filters;
• and
• deduplicate the fixed-size sub-chunks.

A computer readable memory device with instructions stored thereon to perform multi-level inline data deduplication, the instructions comprising:

• fingerprinting, by a deduplication application being executed a physical server of a datacenter, data to be deduplicated to generate fingerprinted content;
• dividing, by a chunk manager of the deduplication application being executed by the physical server of the datacenter, the data into variable-size chunks by an employment of a content dependent chunking (CDC) algorithm based on Rabin fingerprinting to the data, wherein a size of each of the variable-size chunks depends on a size of corresponding fingerprinted content;
• dividing, by the chunk manager of the deduplication application being executed by the physical server of the datacenter, each of the variable-size chunks into fixed-size sub-chunks;
• computing, by the deduplication application being executed by the physical server of the datacenter, the fingerprinted content of the fixed-size sub-chunks, wherein each of the fingerprinted content for the fixed-size sub-chunks is a 20-byte hash;
• storing, by the deduplication application being executed by the physical server of the datacenter, the fingerprinted content of the fixed-size sub-chunks in a two-dimensional array of cells, wherein each cell of the two-dimensional array of cells includes an original fingerprint and an array of a number of bits, and wherein the number of bits reduce a search time to locate the corresponding fingerprinted content for the variable-size chunks;
• and
• deduplicating, by the deduplication application being executed by the physical server of the datacenter, the fixed-size sub-chunks.