Merging entries in a deduplication index

A computer program product for performing deduplication for a storage, the computer program product comprising a computer readable storage device having computer readable program code embodied therein that executes to perform operations, the operations comprising:

• maintaining an index of chunk signatures calculated from chunks of data in the data objects in the storage, wherein each index entry includes at least one of the chunk signatures and a reference to the chunk of data from which the signature was calculated;
• selecting entries in the index to merge;
• performing a merge operation on the chunk signatures in the selected entries to calculate a merged signature;
• adding an entry to the index including the merged signature and a reference to the chunks in the storage referenced in the merged selected entries;
• removing the selected entries used to calculate the merged signature from the index;
• and
• using the index of the signatures in deduplication operations when adding a received data object to the storage by:
• processing the received data object to determine chunks of the data in the received data object;
• determining chunk signatures for the determined chunks by processing the data in the determined chunks of the received data object, wherein each determined chunk signature is associated with the chunk processed to determine the signature;
• determining merged signatures by performing the merge operation on the chunk signatures for consecutive of the determined chunks in the received data object, wherein each determined merged signature is associated with the chunks processed to determine the chunks signatures processed to determine the merged signature;
• determining whether any of the chunk signatures and the merged signatures, associated with the chunks in the received data object, match signatures in the index;
• determining whether there are non-matching chunk or merged signatures calculated from the received data object that do not match the signatures in the index;
• and
• storing the chunks associated with the non-matching chunk or merged signatures chunks in the received data object in the storage, wherein chunks in the received data object associated with the determined chunk and merged signatures that match the signatures in the index are not written to the storage.

A system for performing deduplication for a storage, comprising:

• a processor;
• and
• a computer readable storage medium having programmable code executed by the processor to perform operations, the operations comprising:
• maintaining an index of chunk signatures calculated from chunks of data in the data objects in the storage, wherein each index entry includes at least one of the chunk signatures and a reference to the chunk of data from which the signature was calculated;
• selecting entries in the index to merge by determining entries of the entries in the index based on a criteria of the chunks of data referenced in the entries, wherein the criteria comprises at least one of:
• an age of the chunks in the storage, wherein the age comprises a duration of time since the chunk was added or updated in the storage;
• a frequency of Input/Output (I/O) access to the chunks in the storage;
• a frequency at which the chunks in the storage appear in data objects being added to the storage;
• a region of the storage in which the chunks are stored;
• and a type of data in the chunks;
• performing a merge operation on the chunk signatures in the selected entries to calculate a merged signature;
• adding an entry to the index including the merged signature and a reference to the chunks in the storage referenced in the merged selected entries;
• removing the selected entries used to calculate the merged signature from the index, wherein the resulting merged entries and entries not involved in the merging operation reference all the chunks in the storage included in the data objects maintained in the storage;
• and
• using the index of the signatures in deduplication operations when adding data objects to the storage.

A system for performing deduplication for a storage, comprising:

• a processor;
• and
• a computer readable storage medium having programmable code executed by the processor to perform operations, the operations comprising:
• maintaining an index of chunk signatures calculated from chunks of data in the data objects in the storage, wherein each index entry includes at least one of the chunk signatures and a reference to the chunk of data from which the signature was calculated;
• selecting entries in the index to merge;
• performing a merge operation on the chunk signatures in the selected entries to calculate a merged signature;
• adding an entry to the index including the merged signature and a reference to the chunks in the storage referenced in the merged selected entries;
• removing the selected entries used to calculate the merged signature from the index;
• and
• using the index of the signatures in deduplication operations when adding a received data object to the storage by:
• processing the received data object to determine chunks of the data in the received data object;
• determining chunk signatures for the determined chunks by processing the data in the determined chunks of the received data object, wherein each determined chunk signature is associated with the chunk processed to determine the signature;
• determining merged signatures by performing the merge operation on the chunk signatures for consecutive of the determined chunks in the received data object, wherein each determined merged signature is associated with the chunks processed to determine the chunks signatures processed to determine the merged signature;
• determining whether any of the chunk signatures and the merged signatures, associated with the chunks in the received data object, match signatures in the index;
• determining whether there are non-matching chunk or merged signatures calculated from the received data object that do not match the signatures in the index;
• and
• storing the chunks associated with the non-matching chunk or merged signatures chunks in the received data object in the storage, wherein chunks in the received data object associated with the determined chunk and merged signatures that match the signatures in the index are not written to the storage.

A method for performing deduplication for a storage, comprising:

• maintaining an index of chunk signatures calculated from chunks of data in the data objects in the storage, wherein each index entry includes at least one of the chunk signatures and a reference to the chunk of data from which the signature was calculated;
• selecting entries in the index to merge by determining entries of the entries in the index based on a criteria of the chunks of data referenced in the entries, wherein the criteria comprises at least one of:
• an age of the chunks in the storage, wherein the age comprises a duration of time since the chunk was added or updated in the storage;
• a frequency of Input/Output (I/O) access to the chunks in the storage;
• a frequency at which the chunks in the storage appear in data objects being added to the storage;
• a region of the storage in which the chunks are stored;
• and a type of data in the chunks;
• performing a merge operation on the chunk signatures in the selected entries to calculate a merged signature;
• adding an entry to the index including the merged signature and a reference to the chunks in the storage referenced in the merged selected entries;
• removing the selected entries used to calculate the merged signature from the index, wherein the resulting merged entries and entries not involved in the merging operation reference all the chunks in the storage included in the data objects maintained in the storage;
• and
• using the index of the signatures in deduplication operations when adding data objects to the storage.

A method for performing deduplication for a storage, comprising:

• maintaining an index of chunk signatures calculated from chunks of data in the data objects in the storage, wherein each index entry includes at least one of the chunk signatures and a reference to the chunk of data from which the signature was calculated;
• selecting entries in the index to merge;
• performing a merge operation on the chunk signatures in the selected entries to calculate a merged signature;
• adding an entry to the index including the merged signature and a reference to the chunks in the storage referenced in the merged selected entries;
• removing the selected entries used to calculate the merged signature from the index;
• and
• using the index of the signatures in deduplication operations when adding a received data object to the storage by:
• processing the received data object to determine chunks of the data in the received data object;
• determining chunk signatures for the determined chunks by processing the data in the determined chunks of the received data object, wherein each determined chunk signature is associated with the chunk processed to determine the signature;
• determining merged signatures by performing the merge operation on the chunk signatures for consecutive of the determined chunks in the received data object, wherein each determined merged signature is associated with the chunks processed to determine the chunks signatures processed to determine the merged signature;
• determining whether any of the chunk signatures and the merged signatures, associated with the chunks in the received data object, match signatures in the index;
• determining whether there are non-matching chunk or merged signatures calculated from the received data object that do not match the signatures in the index;
• and
• storing the chunks associated with the non-matching chunk or merged signatures chunks in the received data object in the storage, wherein chunks in the received data object associated with the determined chunk and merged signatures that match the signatures in the index are not written to the storage.