Light-weight index deduplication and hierarchical snapshot replication

A system comprising:

• one or more processors of a machine;
• a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
• generating first index data for a first dataset comprising a first set of segments, the first index data comprising a first full fingerprint for each segment in the first set of segments and a first shortened fingerprint for each segment in the first set of segments, each first shortened fingerprint for a given segment being a shortened version of a corresponding first full fingerprint of the given segment;
• generating second index data for a second dataset comprising a second set of segments, the second index data comprising a second full fingerprint for each segment in the second set of segments and a second shortened fingerprint for each segment in the second set of segments, each second shortened fingerprint for a given segment being a shortened version of a corresponding second full fingerprint of the given segment;
• and
• deduplicating the first dataset and the second dataset by removing duplicate aligned segments using full fingerprints followed by removing duplicate non-aligned segments using shortened fingerprints, the duplicate non-aligned segments being remaining segments not removed as aligned segments, the duplicate aligned segment having a same position in the first dataset and the second dataset, the duplicate non-aligned segments having different positions in the first dataset and the second dataset.

A machine-readable storage device embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:

• generating first index data for a first dataset comprising a first set of segments, the first index data comprising a first full fingerprint for each segment in the first set of segments and a first shortened fingerprint for each segment in the first set of segments, each first shortened fingerprint for a given segment being a shortened version of a corresponding first full fingerprint of the given segment;
• generating second index data for a second dataset comprising a second set of segments, the second index data comprising a second full fingerprint for each segment in the second set of segments and a second shortened fingerprint for each segment in the second set of segments, each second shortened fingerprint for a given segment being a shortened version of a corresponding second full fingerprint of the given segment;
• and
• deduplicating the first dataset and the second dataset by removing duplicate aligned segments using full fingerprints followed by removing duplicate non-aligned segments using shortened fingerprints, the duplicate non-aligned segments being remaining segments not removed as aligned segments, the duplicate aligned segment having a same position in the first dataset and the second dataset, the duplicate non-aligned segments having different positions in the first dataset and the second dataset.