System and method for indexing a data item in a data storage system

A non-transitory computer-readable medium having processor-executable instructions stored thereon for indexing a data item in a data storage system, wherein the processor-executable instructions, when executed, facilitate performance of the following:

• dividing the data item into one or more large blocks;
• dividing each large block into a plurality of small blocks;
• calculating a strong hash value for each of the small blocks of a respective large block and storing a respective list of the strong hash values calculated for the respective large block with a pointer to a location of the respective large block;
• from the respective list of strong hash values calculated for the respective large block, selecting one or more representative hash values for the respective large block;
• and
• compiling a sparse index comprising an entry for each large block, wherein a respective entry for the respective large block is based on the one or more representative hash values selected for the respective large block and a pointer to the respective list of strong hash values for the respective large block.

A data storage system for indexing a data item, comprising:

• one or more memories having processor-executable instructions stored thereon;
• and
• one or more processors configured to execute the processor-executable instructions to facilitate performance of the following:
• dividing the data item into one or more large blocks;
• dividing each large block into a plurality of small blocks;
• calculating a strong hash value for each of the small blocks of a respective large block and storing a respective list of the strong hash values calculated for the respective large block with a pointer to a location of the respective large block;
• from the respective list of strong hash values calculated for the respective large block, selecting one or more representative hash values for the respective large block;
• and
• compiling a sparse index comprising an entry for each large block, wherein a respective entry for the respective large block is based on the one or more representative hash values selected for the respective large block and a pointer to the respective list of strong hash values for the respective large block.