Data reduction indexing

An article of manufacture, comprising:

• a computer readable medium storing computer executable instructions that when executed by a computer control the computer to perform a method, the method comprising:
• indexing, in a global index, a unique chunk processed by a data de-duplicator;
• and
• selectively indexing, in a temporal index, a relationship chunk processed by the data de-duplicator, where the relationship chunk is a chunk that is related to another chunk processed by the data de-duplicator by one or more of, sequence, storage location, and similarity hash value;
• where indexing the unique chunk comprises:
• storing a location context in a selected partition in the global index, the location context comprising a hash value computed from the unique chunk and information identifying where the unique chunk is stored in a storage system accessible to the de-duplicator;
• and
• updating an expedited data structure associated with the selected partition in the global index to include the hash value computed from the unique chunk;
• and
• where selectively indexing the relationship chunk comprises:
• storing a hash value computed from the relationship chunk in one or more of, a partitioned sequence index, a partitioned location index, and a partitioned relation index;
• and
• updating one or more expedited data structures associated with the one or more partitions in the one or more indexes into which the hash value is stored to include the hash value computed from the relationship chunk.

A memory for storing data for access by an application being executed on a data processing system, comprising:

• a data structure stored in the memory, the data structure storing information used by the application and including:
• a global index storing a complete set of location contexts associated with chunks processed by a data de-duplicator;
• and
• a temporal index storing an incomplete set of location contexts associated with chunks processed by the data de-duplicator;
• where the global index is a hash tree whose primary key is a hash field, where the global index is partitioned, and where the global index is protected by at least one expedited data filter;
• and
• where the temporal index comprises:
• a partitioned sequence index arranged as a hash tree and protected by an expedited data filter;
• a partitioned location index arranged as a hash tree and protected by an expedited data filter;
• and
• a partitioned relation index arranged as a hash tree and protected by an expedited data filter,
• and
• where the temporal index stores an ordered set of optimizations configured to facilitate resolving a duplicate determination for the data de-duplicator without accessing the global index.