Detecting missing segments and identifying files impacted by the missing segments using a single pass distributed scan

A non-transitory computer readable medium comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a method for managing data in a storage system, the method comprising:

• partitioning Level 0 (L0) segments of a segment tree to generate buckets;
• assigning a portion of the buckets to a first storage node of the storage system;
• collecting a partition of the L0 segments assigned to the first storage node as child segments;
• identifying:
• a first set of child segments of the child segments that maps to the portion of the buckets of the first storage node, and
• a second set of child segments of the child segments that maps to a bucket of a second storage node of the storage system;
• storing:
• each of the first set of child segments to a log-structured merge (LSM) tree of the first storage node by following a child-segment:
• parent-segment pair format,
• the second set of child segments to a buffer that is mapped to the second storage node;
• sending, once the buffer is filled, the second set of child segments to the second storage node;
• receiving a third set of child segments that maps to the portion of the buckets from the second storage node;
• storing the third set of child segments to the LSM tree by following the format;
• comparing the first set of child segments and the third set of child segments of the LSM tree against second child segments of a key-value store (KVS) listed for an L0 of the segment tree;
• making, based on the comparing, a determination that a mismatch occurred,
• wherein, because a fourth set of child segments of the LSM tree could not be found in the KVS, the mismatch occurred;
• and
• logging, based on the determination, the fourth set of child segments as missing segments.

A system for managing data in a storage system, the system comprising:

• a processor comprising circuitry;
• memory comprising instructions, which when executed by the processor perform a method, the method comprising:
• partitioning Level 0 (L0) segments of a segment tree to generate buckets;
• assigning a portion of the buckets to a first storage node of the storage system;
• collecting a partition of the L0 segments assigned to the first storage node as child segments;
• identifying:
• a first set of child segments of the child segments that maps to the portion of the buckets of the first storage node, and
• a second set of child segments of the child segments that maps to a bucket of a second storage node of the storage system;
• storing:
• each of the first set of child segments to a log-structured merge (LSM) tree of the first storage node by following a child-segment:parent-segment pair format,
• the second set of child segments to a buffer that is mapped to the second storage node;
• sending, once the buffer is filled, the second set of child segments to the second storage node;
• receiving a third set of child segments that maps to the portion of the buckets from the second storage node;
• storing the third set of child segments to the LSM tree by following the format;
• comparing the first set of child segments and the third set of child segments of the LSM tree against second child segments of a key-value store (KVS) listed for an L0 of the segment tree;
• making, based on the comparing, a determination that a mismatch occurred,
• wherein, because a fourth set of child segments of the LSM tree could not be found in the KVS, the mismatch occurred;
• and
• logging, based on the determination, the fourth set of child segments as missing segments.