Systems and methods for distributed data deduplication

A system for distributed data deduplication, the system comprising:

• an identification module programmed to:
• identify a deduplicated data system, the deduplicated data system comprising a plurality of nodes, wherein each node within the plurality of nodes is configured to deduplicate data stored on the node;
• identify a data object to store within the deduplicated data system;
• a hashing module programmed to generate a similarity hash of the data object, the similarity hash representing a probabilistic dimension-reduction of the data object;
• a selection module programmed to:
• determine, based on at least one characteristic of the data object, that block-level deduplication is more efficient for the data object that file-level deduplication;
• select a target node from the plurality of nodes that uses block-level deduplication instead of a candidate node that uses file-level deduplication based at least in part on the similarity hash and at least in part on a determination that the target node uses block-level deduplication instead of file-level deduplication;
• a routing module programmed to route the data object for storage on the target node based on the selection of the target node;
• at least one processor configured to execute the identification module, the hashing module, the selection module, and the routing module.

A non-transitory computer-readable-storage medium comprising one or more computer-executable instructions that are executed by at least one processor of a computing device and cause the computing device to:

• identify a deduplicated data system, the deduplicated data system comprising a plurality of nodes, wherein each node within the plurality of nodes is configured to deduplicate data stored on the node;
• identify a data object to store within the deduplicated data system;
• generate a similarity hash of the data object, the similarity hash representing a probabilistic dimension-reduction of the data object;
• determine, based on at least one characteristic of the data object, that block-level deduplication is more efficient for the data object that file-level deduplication;
• select a target node from the plurality of nodes that uses block-level deduplication instead of a candidate node that uses file-level deduplication based at least in part on the similarity hash and at least in part on a determination that the target node uses block-level deduplication instead of file-level deduplication;
• route the data object for storage on the target node based on the selection of the target node.