Distributed data deduplication in a grid of processors

A system for distributed data deduplication in a grid of processors, comprising:

• an ingesting processor;
• an owning processor in communication with the ingesting processor;
• and
• owning processors of calculated identical data sections, in communication with the processor and the owning processor in the grid of processors;
• wherein the ingesting processor:
• receives input data, and partitions the input data into a plurality of similarity units,
• calculates a corresponding deduplication metadata slice and the owning processor for one of the similarity units;
• wherein the owning processor comprises a first processor in the grid of processors responsible for storing data associated with the one of the similarity units and the ingesting processor comprises a second processor responsible for receiving the input data,
• sends a representative value and corresponding digest values of the similarity unit to the owning processor;
• the owning processor:
• searches for the representative value in the corresponding deduplication metadata slice, and
• sends a specification and the owning processors of calculated identical data sections to the ingesting processor;
• and
• the owning processors of the calculated identical data sections:
• receive nominal information of the calculated identical data sections sent by the ingesting processor, and
• respond to the processor with information of storage blocks storing the identical data sections;
• wherein a deduplication operation is performed on the storage blocks by the owning processor of the calculated identical data sections to reduce redundantly stored data in the grid of processors;
• for the similarity unit, calculates at least one of a deduplication blocks boundaries, the corresponding digest values and the representative value;
• wherein the processor, pursuant to calculating the corresponding deduplication metadata slice and owning processor, uses a consistent hashing method applied to the representative value of the similarity unit;
• wherein the processor sends the representative value and the corresponding digest values of the similarity unit to the owning processor using a single message for several of the plurality of similarity units mapping to a same owning processor.

A computer program product for distributed data deduplication in a grid of processors, the computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

• an executable portion that receives input data on an ingesting processor, and partitions the input data into a plurality of similarity units;
• an executable portion that calculates a corresponding deduplication metadata slice and owning processor for one of the similarity units;
• wherein the owning processor comprises a first processor in the grid of processors responsible for storing data associated with the one of the similarity units and the ingesting processor comprises a second processor responsible for receiving the input data;
• an executable portion that sends a representative value and corresponding digest values of the similarity unit to the owning processor;
• an executable portion that uses the owning processor to search for the representative value in the corresponding deduplication metadata slice;
• an executable portion that uses the owning processor to send a specification and owning processors of calculated identical data sections to the ingesting processor;
• an executable portion that uses the ingesting processor to send nominal information of the calculated identical data sections to the owning processors of the calculated identical data sections;
• and
• an executable portion that uses the owning processors of the calculated identical data sections to respond to the ingesting processor with information of physical storage blocks storing the identical data sections;
• wherein a deduplication operation is performed on the storage blocks by the owning processor of the calculated identical data sections to reduce redundantly stored data in the grid of processors;
• an executable portion that, for the similarity unit, calculates at least one of deduplication blocks boundaries, the corresponding digest values and the representative value;
• an executable portion that calculates the corresponding deduplication metadata slice and owning processor using a consistent hashing method applied to the representative value of the similarity unit;
• an executable portion that sends the representative value and the corresponding digest values of the similarity unit to the owning processor using a single message for several of the plurality of similarity units mapping to a same owning processor.