Use of cluster-level redundancy within a cluster of a distributed storage management system to address node-level errors

A non-transitory machine readable medium storing instructions, which when executed by one or more hardware processors of a cluster of a plurality of nodes representing a distributed storage system, cause the distributed storage system to:

• identify a list of missing block identifiers (IDs) that are in use for one or more volumes associated with a first node of the plurality of nodes but that are missing from an instance of a key-value (KV) store of the first node by performing a data integrity check on the instance of the KV store;
• and
• after identifying the list of missing block IDs, avoiding treating the first node as failed and instead performing a resynchronization process by, for each block ID in the list of block IDs:
• reading a data block corresponding to the block ID from a second node of the plurality of nodes that maintains a redundant data block corresponding to the block ID;
• and
• restoring the block ID within the KV store by writing the redundant data block to the first node.

A distributed storage system comprising:

• a plurality of nodes operating as a cluster and including one or more hardware processors and instructions that when executed by the one or more hardware processors cause the distributed storage system to:
• manage, within an instance of a key-value (KV) store of a first node of the plurality of nodes, storage of data blocks as values and corresponding block identifiers (IDs) as keys;
• and
• avoid treating the first node as failed and instead performing a resynchronization process by, for each block ID in a list of missing block IDs that are in use for one or more volumes associated with the first node but that are missing from the instance of the KV store:
• reading a data block corresponding to the block ID from a second node of the plurality of nodes that maintains a redundant data block corresponding to the block ID;
• and
• restoring the block ID within the KV store by writing the redundant data block to the first node.