Optmizing data deduplication

A system comprising:

• a processor;
• and
• a memory comprising code stored thereon, that when executed, performs a method of processing data comprising:
• receiving a first candidate block including a plurality of uniformly-sized sub-blocks;
• performing partial deduplication processing of the first candidate block, wherein the partial deduplication processing includes:
• partially deduplicating at least one sub-block of the first candidate block;
• and
• creating a first entry in a deduplication database for the first candidate block, wherein the first entry includes a first digest of the first candidate block and the first entry denotes a potential target block having the first digest, and wherein the first entry includes a counter that tracks a number of missed full block deduplications between the potential target block and subsequently processed candidate blocks;
• receiving a second candidate block subsequent to performing partial deduplication processing for the first candidate block;
• and
• performing first processing to determine whether to perform promotion processing for the first entry, wherein said promotion processing promotes the potential target block, having the first digest of the first entry, to a new target block.

A computer readable medium comprising code stored thereon that, when executed, performs a method of processing data comprising:

• receiving a first candidate block including a plurality of uniformly-sized sub-blocks;
• performing partial deduplication processing of the first candidate block, wherein the partial deduplication processing includes:
• partially deduplicating at least one sub-block of the first candidate block;
• and
• creating a first entry in a deduplication database for the first candidate block, wherein the first entry includes a first digest of the first candidate block and the first entry denotes a potential target block having the first digest, and wherein the first entry includes a counter that tracks a number of missed full block deduplications between the potential target block and subsequently processed candidate blocks;
• receiving a second candidate block subsequent to performing partial deduplication processing for the first candidate block;
• and
• performing first processing to determine whether to perform promotion processing for the first entry, wherein said promotion processing promotes the potential target block, having the first digest of the first entry, to a new target block.