System and methods for removing obsolete data in a distributed system of hybrid storage and compute nodes

A computerized method that garbage collects in a distributed storage system, the method comprising:

• generating a live object map of live objects stored on the distributed storage system in a plurality of block segments distributed across a plurality of storage controller servers, wherein the live object map is a probabilistic data structure storing a list of valid objects;
• scanning the plurality of block segments to generate segment summary statistics, wherein the segment summary statistics indicates the number of live objects stored in the plurality of block segments;
• and
• compacting each of the plurality of block segments that have a low utilization based on the segment summary statistics.

A distributed storage system to garbage collect in a distributed storage system, the distributed storage system comprising:

• an interconnection network;
• and
• a plurality of storage controllers, interconnected by the interconnection network, wherein each of the plurality of storage controllers includes,
• a garbage collection module that performs garbage collection in the distributed storage system, the garbage collection module including,
• a live object map module that generates a live object map of live objects stored on the distributed storage system in a plurality of block segments distributed across a plurality of storage controller servers, wherein the live object map is a probabilistic data structure storing a list of valid objects;
• a segment cleaner, coupled to the live object map, that scans the plurality of block segments to generate segment summary statistics, wherein the segment summary statistics indicates the number of live objects stored in the plurality of block segments;
• and
• segment compacter, coupled to the segment cleaner, that compacts each of the plurality of block segments that have a low utilization based on the segment summary statistics.

A non-transitory machine-readable medium having executable instructions to cause one or more processing units to perform a method to garbage collect in a distributed storage system, the method comprising:

• generating a live object map of live objects stored on the distributed storage system in a plurality of block segments distributed across a plurality of storage controller servers, wherein the generating the live object includes,
• traversing a metadata tree, and
• adding an object to the live object map for each entry in the metadata tree;
• scanning the plurality of block segments to generate segment summary statistics, wherein the segment summary statistics indicates the number of live objects stored in the plurality of block segments;
• and
• compacting each of the plurality of block segments that have a low utilization based on the segment summary statistics.

A non-transitory machine-readable medium having executable instructions to cause one or more processing units to perform a method to garbage collect in a distributed storage system, the method comprising:

• generating a live object map of live objects stored on the distributed storage system in a plurality of block segments distributed across a plurality of storage controller servers, wherein the generating the live object includes,
• adding list of objects being deduplicated to the live object map;
• scanning the plurality of block segments to generate segment summary statistics, wherein the segment summary statistics indicates the number of live objects stored in the plurality of block segments;
• and
• compacting each of the plurality of block segments that have a low utilization based on the segment summary statistics.

A non-transitory machine-readable medium having executable instructions to cause one or more processing units to perform a method to garbage collect in a distributed storage system, the method comprising:

• generating a live object map of live objects stored on the distributed storage system in a plurality of block segments distributed across a plurality of storage controller servers;
• scanning the plurality of block segments to generate segment summary statistics, wherein the segment summary statistics indicates the number of live objects stored in the plurality of block segments, wherein the scanning includes,
• building a keylist for the segment;
• and
• verifying the number of live keys in the live object map;
• and
• compacting each of the plurality of block segments that have a low utilization based on the segment summary statistics.