Garbage collection through namespace metadata indexing using a key value store on a cluster filesystem

A method of reducing data stored in a cluster file system that maps a file name to an INODE data structure that encapsulates metadata of the file, the method comprising:

• deploying a key value store (KVS) database having key:value elements and utilizing a key value pair (KVP) comprising an INODE type to store metadata as INODE information, and a DIRENT type to store file names as DIRENT information;
• defining a logical table (LT) of the KVS having a logical table ID (LTID), and a logical table group (LTG) of the KVS containing the LT and having a logical table group ID (LTGID);
• storing the file ID as a key element in the KVS, where a corresponding value comprises metadata and associated information for the file for the INODE KVP;
• initiating a garbage collection (GC) process to remove metadata associated with snapshot copies of files of the cluster file system, each snapshot copy of the snapshot copies having a corresponding and unique snapshot ID (SID) and taken at a specific time from old to new;
• performing a pair-wise comparison of snapshot copies from oldest to newest snapshot to find a deleted snapshot;
• deleting all present INODE information for the deleted snapshot;
• and
• performing a conditional deletion of DIRENT information based on whether DIRENTs are saved or exist in a different SID level snapshot, wherein the cluster file system comprises a network of individual nodes having a log-structured file system implemented in a Data Domain Filesystem, and wherein the namespace contains file metadata implemented as a layered KVS, and further wherein the cluster file system comprises a Santorini network processing containerized data utilizing a Kubernetes-based framework, and wherein the Santorini network comprises part of a Data Domain deduplication backup system performing backup and restore operations for the plurality of nodes and stores file system metadata in the KVS.

A system reducing data stored in a cluster file system that maps a file name to an INODE data structure that encapsulates metadata of the file, and comprising:

• a network of individual nodes having a log-structured file system implemented in a Data Domain Filesystem, and wherein the namespace contains file metadata implemented as a layered KVS, and further wherein the cluster file system comprises a Santorini network processing containerized data utilizing a Kubernetes-based framework, and wherein the Santorini network comprises part of a Data Domain deduplication backup system performing backup and restore operations for the plurality of nodes and stores file system metadata in a key-value store (KVS);
• a first storage containing a KVS database having key:value elements and utilizing a key value pair (KVP) comprising an INODE type to store metadata and a DIRENT type to store file names;
• a memory storing a logical table (LT) of the KVS having a logical table ID (LTID), and second defining a logical table group (LTG) of the KVS containing the LT and having a logical table group ID (LTGID), and encoding a directory location of the file through a file ID specifying a parent ID (PID) and child ID (CID);
• a the second storage storing the file ID as a key element in the KVS, where a corresponding value comprises metadata and associated information for the file for the INODE KVP;
• a garbage collection (GC) component configured to remove metadata associated with snapshot copies of files of the cluster file system, each snapshot copy of the snapshot copies having a corresponding and unique snapshot ID (SID) and taken at a specific time from old to new;
• and
• a comparator performing a pair-wise comparison of snapshot copies from oldest to newest snapshot to find a deleted snapshot, and wherein the GC component deletes all present INODE information for the deleted snapshot, and conditionally deletes DIRENT information based on whether DIRENTs are saved or exist in a different SID level snapshot.