Storing data deduplication metadata in a grid of processors

A system for storing data deduplication metadata, comprising:

• a processor, operable in a grid of processors, that:
• assigns each of a plurality of slices of deduplication metadata to be stored in the grid of processors;
• wherein each slice of the plurality of slices includes at least one of a slice of a similarity index, and groups of digests corresponding to those of a plurality of input representative values in the slice of the similarity index;
• and wherein the grid of processors further includes at least an owning processor comprising a first processor in the grid of processors responsible for storing data associated with the slices of deduplication metadata and an ingesting processor comprising a second processor responsible for receiving the data as the data is input into the grid of processors;
• uses a hashing method to map between a plurality of input representative values and the plurality of slices of deduplication metadata;
• performs a deduplication operation by the owning processor or an additional processor using the slices of deduplication metadata on the input data to reduce redundantly stored data in the grid of processors;
• calculates the mapping between the plurality of input representative values and the plurality of slices of deduplication metadata using the hashing method to produce a consistent ranking on the plurality of slices of deduplication metadata for the plurality of input representative values;
• and stores one of the plurality of input representative values and groups of digests corresponding to the one of the plurality of input representative values in K metadata slices, where the K metadata slices have a highest rank value.

A computer program product for storing data deduplication metadata in a grid of processors, the computer program product comprising a computer-readable storage medium having computer readable program code portions stored therein, the computer-readable program code portions comprising:

• an executable portion that assigns each of a plurality of slices of deduplication metadata to be stored by a corresponding processor in a grid of processors;
• wherein each slice of the plurality of slices includes at least one of a slice of a similarity index and groups of digests corresponding to those of a plurality of representative values in the slice of the similarity index;
• and wherein the grid of processors further includes at least an owning processor comprising a first processor in the grid of processors responsible for storing data associated with the slices of deduplication metadata and an ingesting processor comprising a second processor responsible for receiving the data as the data is input into the grid of processors;
• an executable portion that uses a hashing method to map between a plurality of input representative values and the plurality of slices of deduplication metadata;
• an executable portion that performs a deduplication operation by the owning processor or an additional processor using the slices of deduplication metadata on the input data to reduce redundantly stored data in the grid of processors;
• an executable portion that calculates the mapping between the plurality of input representative values and the plurality of slices of deduplication metadata using the hashing method to produce a consistent ranking on the plurality of slices of deduplication metadata for the plurality of input representative values;
• and an executable portion that stores one of the plurality of input representative values and groups of digests corresponding to the one of the plurality of input representative values in K metadata slices, where the K metadata slices have a highest rank value.