Similarity based deduplication for secondary storage

A method comprising:

• generating a rolling hash value based on a portion of an incoming stream of backup data;
• comparing the rolling hash value with entries stored in a rolling hash index, and in response to matching the rolling hash value with at least one entry in the rolling hash index, generating a strong hash value and determining if a match of the strong hash value exists in a first strong hash index of one of either a first public cloud or a first private cloud, and where the comparison module, in response to a determination that the match does not exist in the first strong hash index, further comparing the strong hash value with entries in a second strong hash index in one of either a second public cloud or a second private cloud;
• and
• storing, in response to a determination that the strong hash value does not match an entry in the first strong hash index or the second strong hash index, the portion of the incoming stream of backup data as new data in at least one of the first public cloud, first private cloud, second public cloud, or second private cloud.

A computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions readable/executable by a processor to cause the processor to execute steps comprising:

• generating, by processor, a rolling hash value based on a portion of an incoming stream of backup data;
• comparing the rolling hash value with entries stored in a rolling hash index, and in response to matching the rolling hash value with at least one entry in the rolling hash index, generating a strong hash value and determining if a match of the strong hash value exists in a first strong hash index of one of either a first public cloud or a first private cloud, and where the comparison module, in response to a determination that the match does not exist in the first strong hash index, further comparing the strong hash value with entries in a second strong hash index in one of either a second public cloud or a second private cloud;
• and
• storing, in response to a determination that the strong hash value does not match an entry in the first strong hash index or the second strong hash index, the portion of the incoming stream of backup data as new data in at least one of the first public cloud, first private cloud, second public cloud, or second private cloud.