System and method for data deduplication

A system for deduplicating data comprising:

• a card operable to receive at least one data block;
• a processor on the card that generates a hash for each data block;
• a first module that determines a processing status for the hash;
• a second module that discards duplicate hashes and their data blocks and writes unique hashes and their data blocks to a computer readable medium.

A system for deduplicating data comprising:

• a computer that includes a computer readable medium and general processor operating thereon;
• a card, coupled to the computer, that includes a processor for performing hashing operations and hardware-assisted compression;
• a virtual block device that is an interface with applications and that is operable to set a block size for deduplication;
• a deduplication block manager that passes blocks to the processor and receives a hash for each block and a compressed block from the processor;
• a truncated hash array that includes an index for each possible truncated hash value and a corresponding entry initially set to a default value wherein when a new hash is processed, the corresponding entry is modified to include a pointer to the new hash and block;
• wherein the card
• receives blocks from the deduplication block manager,
• creates a hash value for each block of data using a hashing algorithm, and
• compresses each block of data using a compression algorithm;
• and
• wherein the deduplication block manager
• receives the hash values and the compressed blocks from the card, and
• creates a truncated hash corresponding to each hash value.