Method and system for distributed garbage collection of deduplicated datasets

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations, the operations comprising:

• determining a set of live fingerprints for each of one or more deduplicated storage containers, wherein for each of the deduplicated storage containers, the set of live fingerprints includes a fingerprint stored on the storage container that is associated with a file in a file system directory structure, wherein the determining of the set of live fingerprints includes
• performing a query into each Bloom filter in a set of Bloom filters to determine whether that Bloom filter is associated with a container identifier (CID) for each fingerprint in a fingerprint index associated with the deduplicated storage containers, and
• inserting each fingerprint and CID pair into the set of live fingerprints, in response to determining that Bloom filter is associated with the CID;
• and
• performing garbage collection operations by copying a storage segment having a fingerprint in the set of live fingerprints from the one or more deduplicated storage containers to one or more new storage containers.

A data processing system, comprising:

• a processor;
• and
• a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations, the operations including
• determining a set of live fingerprints for each of one or more deduplicated storage containers, wherein for each of the deduplicated storage containers, the set of live fingerprints includes a fingerprint stored on the storage container that is associated with a file in a file system directory structure, wherein the determining of the set of live fingerprints includes
• performing a query into each Bloom filter in a set of Bloom filters to determine whether that Bloom filter is associated with a container identifier (CID) for each fingerprint in a fingerprint index associated with the deduplicated storage containers, and
• inserting each fingerprint and CID pair into the set of live fingerprints, in response to determining that Bloom filter is associated with the CID;
• and
• performing garbage collection operations by copying a storage segment having a fingerprint in the set of live fingerprints from the one or more deduplicated storage containers to one or more new storage containers.