Corrupted bucket handling in a peer cluster

A computing device, comprising:

• a processor;
• and
• a non-transitory computer-readable medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:
• receiving, by a cluster manager of a peer cluster comprising a plurality of peer nodes, a notification of a corrupted bucket stored in the peer cluster;
• and
• performing, by the cluster manager, a bucket recovery operation, comprising:
• for each peer node of a subset of the plurality of peer nodes, sending a request to the peer node to report a bucket health metric for a bucket copy corresponding to the corrupted bucket stored on the peer node,
• obtaining, from each peer node of the subset of the plurality of peer nodes, a corresponding bucket health metric comprising a searchable event count of the respective bucket copy stored on the peer node,
• electing, as a canonical bucket copy, a bucket copy having a searchable event count with a highest value, wherein the canonical bucket copy is stored on an elected peer node, and
• sending a replicate command to the elected peer node storing the canonical bucket copy to replicate the canonical bucket copy to the plurality of peer nodes of the peer cluster.

A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• receiving, by a cluster manager of a peer cluster comprising a plurality of peer nodes, a notification of a corrupted bucket stored in the peer cluster;
• and
• performing, by the cluster manager, a bucket recovery operation, comprising:
• for each peer node of a subset of the plurality of peer nodes, sending a request to the peer node to report a bucket health metric for a bucket copy corresponding to the corrupted bucket stored on the peer node,
• obtaining, from each peer node of the subset of the plurality of peer nodes, a corresponding bucket health metric comprising a searchable event count of the respective bucket copy stored on the peer node,
• electing, as a canonical bucket copy, a bucket copy having a searchable event count with a highest value, wherein the canonical bucket copy is stored on an elected peer node, and
• sending a replicate command to the elected peer node storing the canonical bucket copy to replicate the canonical bucket copy to the plurality of peer nodes of the peer cluster.