Consistent deduplicated snapshot generation for a distributed database using optimistic deduplication

A system for generating a consistent deduplicated snapshot of a distributed database using optimistic deduplication, the system comprising:

• one or more computer readable storage media;
• a processing system operatively coupled with the one or more computer readable storage media;
• and
• program instructions stored on the one or more computer readable storage media that, when read and executed by the processing system, direct the processing system to perform operations comprising:
• for each node of a plurality of nodes in the distributed database,
• deduplicating data items that are identified as being stored on the node and owned by the node and
• generating a summary for the node, the summary describing a file in which the data items that are identified as being stored on the node are located;
• identifying from the summaries for each of the nodes whether one or more mistakes occurred during deduplication;
• upon identifying the one or more mistakes, determining one or more corrections for the one or more mistakes;
• and
• generating a consistent deduplicated snapshot for the distributed database comprising the deduplicated data items from each node and the one or more corrections.