Method to keep on-disk physical deduplication index layer up to date

A system comprising:

• a memory;
• and
• processing circuitry configured to execute program instructions out of the memory to:
• provide a plurality of deduplication (“dedupe”) indexes across a memory level and a disk level of the system, the plurality of dedupe indexes including an in-memory dedupe index at the memory level, and an on-disk dedupe index at the disk level, the plurality of dedupe indexes for use in performing data deduplication operations;
• apply a plurality of eviction rules regarding evicting digest entries of the on-disk dedupe index, the plurality of eviction rules avoiding random evictions of digest entries from the on-disk dedupe index;
• in response to applying the plurality of eviction rules, identify a number of digest entries of the on-disk dedupe index as eviction candidates;
• and
• evict the identified eviction candidates from the on-disk dedupe index,
• wherein each of the plurality of dedupe indexes includes a plurality of digest entries, each digest entry including a digest of a stored dataset and a pointer to information pertaining to a location of the stored dataset, and
• wherein the processing circuitry is configured to execute the program instructions out of the memory to:
• apply a first eviction rule including avoiding removing or evicting digest entries for stored datasets pointed to by virtual pointers having a reference count with a first value greater than a predetermined maximum reference count;
• apply a second eviction rule including preferring removing or evicting digest entries for stored datasets pointed to by virtual pointers having the predetermined maximum reference count;
• apply a third eviction rule including preferring removing or evicting digest entries for stored datasets deemed non-deduplicable;
• and
• apply a fourth eviction rule including avoiding removing or evicting “clean” digest entries previously destaged from the in-memory dedupe index.

A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method comprising:

• providing a plurality of deduplication (“dedupe”) indexes across a memory level and a disk level of a storage system, the plurality of dedupe indexes including an in-memory dedupe index at the memory level, and an on-disk dedupe index at the disk level, the plurality of dedupe indexes for use by the storage system in performing data deduplication operations;
• applying a plurality of eviction rules regarding evicting digest entries of the on-disk dedupe index, the plurality of eviction rules avoiding random evictions of digest entries from the on-disk dedupe index;
• in response to applying the plurality of eviction rules, identifying a number of digest entries of the on-disk dedupe index as eviction candidates;
• and
• evicting the identified eviction candidates from the on-disk dedupe index,
• wherein each of the plurality of dedupe indexes includes a plurality of digest entries, each digest entry including a digest of a stored dataset and a pointer to information pertaining to a location of the stored dataset, and
• wherein the applying a plurality of eviction rules includes:
• applying a first eviction rule including avoiding removing or evicting digest entries for stored datasets pointed to by virtual pointers having a reference count with a first value greater than a predetermined maximum reference count;
• applying a second eviction rule including preferring removing or evicting digest entries for stored datasets pointed to by virtual pointers having the predetermined maximum reference count;
• applying a third eviction rule including preferring removing or evicting digest entries for stored datasets deemed non-deduplicable;
• and
• applying a fourth eviction rule including avoiding removing or evicting “clean” digest entries previously destaged from the in-memory dedupe index.