Fully distributed intelligent rebuild

A computer program product for reducing the probability of losing data while in storage, the computer program product comprising:

• one or more computer readable storage media;
• and
• program instructions stored on the one or more computer readable storage media, the program instructions comprising:
• program instructions to determine that a first data replica in a first plurality of data stored on a first device of a distributed storage system is not in sync with a second data replica in a second plurality of data stored on a second device of the distributed storage system, wherein the second plurality of data is a replica of the first plurality of data;
• program instructions to add the first data replica in the first plurality of data to an unsynced queue and adding, by one or more computer processors, a neighbor data of the first plurality of data to a suspect queue, wherein the neighbor data is the first plurality of data not including the first data replica;
• program instructions to increase a priority to check the neighbor data if the neighbor data is already in the suspect queue;
• responsive to determining the neighbor data in the suspect queue is not in sync with any data replica stored to any device, program instructions to add the unsynced neighbor data to the unsynced queue;
• program instructions to determine a priority for recovery, wherein the priority for recovery is based on a vulnerability of the data in the unsynced queue;
• and
• program instructions to recover at least one data replica in the unsynced queue, wherein the recovery is a replacement of the at least one data replica with an intact replica of the at least one replica.

A computer system for reducing the probability of losing data while in storage, the computer system comprising:

• one or more computer processors;
• one or more computer readable storage media;
• and
• program instructions stored on the one or more computer readable storage media for execution by at least one of the one or more computer processors, the program instructions comprising:
• program instructions to determine that a first data replica in a first plurality of data stored on a first device of a distributed storage system is not in sync with a second data replica in a second plurality of data stored on a second device of the distributed storage system, wherein the second plurality of data is a replica of the first plurality of data;
• program instructions to add the first data replica in the first plurality of data to an unsynced queue and adding, by one or more computer processors, a neighbor data of the first plurality of data to a suspect queue, wherein the neighbor data is the first plurality of data not including the first data replica;
• program instructions to increase a priority to check the neighbor data if the neighbor data is already in the suspect queue;
• responsive to determining the neighbor data in the suspect queue is not in sync with any data replica stored to any device, program instructions to add the unsynced neighbor data to the unsynced queue;
• program instructions to determine a priority for recovery, wherein the priority for recovery is based on a vulnerability of the data in the unsynced queue;
• and
• program instructions to recover at least one data replica in the unsynced queue, wherein the recovery is a replacement of the at least one data replica with an intact replica of the at least one replica.