Reducing resource consumption of a similarity index in data deduplication

A system for reducing resource consumption of a similarity index in performing data deduplication, comprising:

• a processor, operable to perform the data deduplication, wherein the processor:
• represents in the similarity index only a latest generation of repository data, wherein a single latest entry is maintained for a representative value in the similarity index, applies implicit deletion in the similarity index such that a similarly index entry is not removed or overwritten until a change with associated data of the similarity index entry is detected,
• maintains a subset of bytes of the representative value in the similarity index entry to reduce an input/output (I/O) load on the similarity index of the system, deduplicates a respective one of a plurality of similarity units with a corresponding similarity unit of a previous snapshot, and examines a deduplication coverage;
• wherein if a deduplication coverage threshold is not met, a similarity search is conducted and the respective one of the plurality of similarity units is deduplicated with a unified set of digest matches combined from the corresponding similarity unit of the previous snapshot and a found similarity unit residing in the similarity index.

A computer program product for reducing resource consumption of a similarity index in a data deduplication system by a processor, the computer program product comprising a computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

• an executable portion that represents in the similarity index only a latest generation of repository data, wherein a single latest entry is maintained for a representative value in the similarity index;
• an executable portion that applies implicit deletion in the similarity index such that a similarly index entry is not removed or overwritten until a change with associated data of the similarity index entry is detected;
• an executable portion that maintains a subset of bytes of the representative value in the similarity index entry to reduce an input/output (I/O) load on the similarity index of the data deduplication system;
• an executable portion that deduplicates a respective one of a plurality of similarity units with a corresponding similarity unit of a previous snapshot;
• and
• an executable portion that examines a deduplication coverage;
• wherein if a deduplication coverage threshold is not met, a similarity search is conducted and the respective one of the plurality of similarity units is deduplicated with a unified set of digest matches combined from the corresponding similarity unit of the previous snapshot and a found similarity unit residing in the similarity index.