Collaborative Multilevel Erasure Coding for Maximizing Durability and Performance of Storage Systems

A computer-implemented method for managing failures in a hierarchical storage system, the method comprising:

• storing data across a plurality of storage devices and storage nodes arranged in a hierarchical structure, wherein each storage node comprises multiple storage devices;
• detecting a failure of a first storage device on a first node;
• determining whether reconstruction of data previously stored on the first storage device is to be performed at storage device level or storage node level based on a redundancy level of data associated with the first storage device, a number of failed storage devices on the first node, and a number of failed storage nodes, each failed storage node including at least one failed storage device, comprising:
• in response to determining that a number of failed storage nodes does not exceed a predetermined threshold, determining that reconstruction is to be performed at the storage node level, wherein reconstruction performed at the storage node level includes reconstructing lost data using data stored on other storage nodes;
• and
• causing the reconstruction of data previously stored on the first storage device to be performed at the storage node level;
• detecting a failure of a second storage device on a second node;
• determining whether reconstruction of data previously stored on the second storage device is to be performed at storage device level or storage node level based on a redundancy level of data associated with the second storage device, a number of failed storage devices on the first node, and a number of failed storage nodes, comprising:
• in response to determining that the number of failed storage nodes exceeds the predetermined threshold, determining that reconstruction is to be performed at the storage device level, wherein reconstruction performed at the storage device level includes reconstructing lost data using data stored on other storage devices within a same node on which failed storage device resides;
• and
• causing the reconstruction of data previously stored on the second storage device to be performed at the storage device level.

A non-transitory computer-readable storage medium storing instructions that when executed by one or more computer processors, cause the one or more computer processors to perform steps comprising:

• storing data across a plurality of storage devices and storage nodes arranged in a hierarchical structure, wherein each storage node comprises multiple storage devices;
• detecting a failure of a first storage device on a first node;
• determining whether reconstruction of data previously stored on the first storage device is to be performed at storage device level or storage node level based on a redundancy level of data associated with the first storage device, a number of failed storage devices on the first node, and a number of failed storage nodes, each failed storage node including at least one failed storage device, comprising:
• in response to determining that a number of failed storage nodes does not exceed a predetermined threshold, determining that reconstruction is to be performed at the storage node level, wherein reconstruction performed at the storage node level includes reconstructing lost data using data stored on other storage nodes;
• and
• causing the reconstruction of data previously stored on the first storage device to be performed at the storage node level;
• detecting a failure of a second storage device on a second node;
• determining whether reconstruction of data previously stored on the second storage device is to be performed at storage device level or storage node level based on a redundancy level of data associated with the second storage device, a number of failed storage devices on the first node, and a number of failed storage nodes, comprising:
• in response to determining that the number of failed storage nodes exceeds the predetermined threshold, determining that reconstruction is to be performed at the storage device level, wherein reconstruction performed at the storage device level includes reconstructing lost data using data stored on other storage devices within a same node on which failed storage device resides;
• and
• causing the reconstruction of data previously stored on the second storage device to be performed at the storage device level.

A computing system comprising:

• one or more computer processors;
• and
• a non-transitory computer readable storage medium storing instructions that when executed by the one or more computer processors, cause the one or more computer processors to perform steps comprising computer-implemented method, comprising:
• storing data across a plurality of storage devices and storage nodes arranged in a hierarchical structure, wherein each storage node comprises multiple storage devices;
• detecting a failure of a first storage device on a first node;
• determining whether reconstruction of data previously stored on the first storage device is to be performed at storage device level or storage node level based on a redundancy level of data associated with the first storage device, a number of failed storage devices on the first node, and a number of failed storage nodes, each failed storage node including at least one failed storage device, comprising:
• in response to determining that a number of failed storage nodes does not exceed a predetermined threshold, determining that reconstruction is to be performed at the storage node level, wherein reconstruction performed at the storage node level includes reconstructing lost data using data stored on other storage nodes;
• and
• causing the reconstruction of data previously stored on the first storage device to be performed at the storage node level;
• detecting a failure of a second storage device on a second node;
• determining whether reconstruction of data previously stored on the second storage device is to be performed at storage device level or storage node level based on a redundancy level of data associated with the second storage device, a number of failed storage devices on the first node, and a number of failed storage nodes, comprising:
• in response to determining that the number of failed storage nodes exceeds the predetermined threshold, determining that reconstruction is to be performed at the storage device level, wherein reconstruction performed at the storage device level includes reconstructing lost data using data stored on other storage devices within a same node on which failed storage device resides;
• and
• causing the reconstruction of data previously stored on the second storage device to be performed at the storage device level.