Reference Set Construction for Data Deduplication

A method comprising:

• receiving, at a non-transitory storage device, a data stream including one or more data blocks;
• identifying a first chunk stored within the non-transitory storage device;
• retrieving a first local reference set from the first chunk;
• retrieving a global reference set from the non-transitory storage device;
• evaluating a performance of the first local reference set and the global reference set;
• selecting one of the first local reference set and the global reference set based on the evaluated performance;
• deduplicating each of the one or more data blocks using the selected reference set;
• and
• associating the deduplicated data blocks with the selected reference set.

A data storage system comprising:

• means for receiving a data stream including one or more data blocks at a non-transitory storage device;
• means for identifying a first chunk stored within the non-transitory storage device;
• means for retrieving a first local reference set from the first chunk;
• means for retrieving a global reference set from the non-transitory storage device;
• means for evaluating a performance of the first local reference set and the global reference set;
• means for selecting one of the first local reference set and the global reference set based on the evaluated performance;
• means for deduplicating each of the one or more data blocks using the selected reference set;
• and
• means for associating the deduplicated data blocks with the selected reference set.