Systems and methods for facilitating file reconstruction and restoration in data storage systems where a RAID-X format is implemented at a file level within a plurality of storage devices

In a distributed, object-based data storage system that includes a plurality of object-based storage devices, and one or more nodes that access virtualized objects from the object-based storage devices, wherein each of the virtualized objects is stored as an aggregation of multiple component objects, each stored on one of the object-based storage devices, and wherein a RAID-X format is implemented at a file level within the plurality of object-based storage devices, a method for reconstruction comprising:

• (a) in response to a failure that renders inaccessible at least one portion of an object-based storage device, retrieving a map for at least one virtualized object having at least one component stored on the at least one portion;
• wherein the map includes a list of storage devices where components of the at least one virtualized object reside;
• (b) using the map to determine whether a number of lost or damaged components associated with the at least one virtualized object exceed a maximum limit beyond which reconstruction of an object stored using the file level implementation of the RAID-X format is not possible;
• and
• (c) if the maximum limit for reconstruction of an object stored using the RAID-X format has not been exceeded, reconstructing the at least one virtualized object;
• wherein the data storage system continues to service data storage and access requests, with reconstruction of the at least one virtualized object being performed as a background operation, after a number of device failures has occurred which would have exceeded the maximum limit of storage device failures for reconstruction had the virtualized object been stored using a volume level implementation of the RAID-X format;
• and
• wherein said at least one virtualized object contains data from one or more logical files, or a portion of a logical file.

In a data storage system that includes a plurality of storage devices, and one or more nodes that access files from the storage devices, wherein a RAID-X format is implemented at a file level within the plurality of storage devices, a method for file reconstruction comprising:

• (a) in response to a failure of one or more of the storage devices, retrieving a map for at least one file having at least one component stored on the one or more failed storage devices;
• wherein the map includes a list of storage devices where components of the at least one file reside;
• (b) using the map to determine whether a number of lost or damaged components associated with the at least one file exceed a maximum limit beyond which reconstruction of a file stored using the file level implementation of the RAID-X format is not possible, and identifying one or more lost or damaged files where the maximum limit for reconstruction of the file stored using the file level implementation of the RAID-X format has been exceeded;
• wherein the data storage system continues to service data storage and access requests, with step (b) being performed as a background operation, after a number of device failures has occurred which would have exceeded the maximum limit of storage device failures for reconstruction had the file been stored using a volume level implementation of the RAID-X format;
• and
• wherein step (b) further comprises at least one of:
• (i) providing a list of the one or more lost or damaged files to an administrator in order to facilitate a manual restore from backup of the one or more lost or damaged files;
• and
• (ii) for each of the one or more lost or damaged files, automatically retrieving a most recent version of the file from a backup location.

In a distributed, object-based data storage system that includes a plurality of object-based storage devices, and one or more nodes that access virtualized objects from the object-based storage devices, wherein each of the virtualized objects is stored as an aggregation of multiple component objects, each stored on one of the object-based storage devices, and wherein a RAID-X format is implemented at a file level within the plurality of object-based storage devices, a method for reconstruction comprising:

• (a) in response to a failure of one or more of the storage devices, retrieving a map for at least one virtualized object having at least one component stored on the one or more failed storage devices;
• wherein the map includes a list of storage devices where components of the at least one virtualized object reside;
• (b) using the map to determine whether a number of lost or damaged components associated with the at least one virtualized object exceed a maximum limit beyond which reconstruction of a virtualized object stored using the file level implementation of the RAID-X format is not possible, and identifying one or more lost or damaged virtualized objects where the maximum limit for reconstruction of the virtualized objects stored using the file level implementation of the RAID-X format has been exceeded;
• wherein the data storage system continues to service data storage and access requests, with step (b) being performed as a background operation, after a number of device failures has occurred which would have exceeded the maximum limit of storage device failures for reconstruction had the at least one virtualized object been stored using a volume level implementation of the RAID-X format;
• and
• wherein step (b) further comprises at least one of:
• (i) providing a list of the one or more lost or damaged virtualized objects to an administrator in order to facilitate a manual restore from backup of the one or more lost or damaged virtualized objects;
• and
• (ii) for each of the one or more lost or damaged virtualized objects, automatically retrieving a most recent version of the virtualized object from a backup location;
• wherein said at least one virtualized object contains data from one or more logical files, or a portion of a logical file.

In a data storage system that includes a plurality of storage devices, and one or more nodes that access files from the storage devices, wherein a RAID-X format is implemented at a file level within the plurality of storage devices, a method for reconstruction of only a part of a file, comprising:

• (a) in response to a failure that renders inaccessible at least one portion of a storage device, retrieving a map for a file having at least one component stored on the at least one portion;
• wherein the map includes a list of storage devices where components of part of the file reside wherein the part corresponds to less than all parts of the file;
• (b) using the map to determine whether a number of lost or damaged components associated with the part of the file exceed a maximum limit beyond which reconstruction of the part of the file stored using the file level implementation of the RAID-X format is not possible;
• and
• (c) if the maximum limit for reconstruction using the RAID-X format has not been exceeded, reconstructing the part of the file while leaving remaining parts of the file intact;
• wherein the data storage system continues to service data storage and access requests, with reconstruction of the part of the file being performed as a background operation, after a number of device failures has occurred which would have exceeded the maximum limit of storage device failures for reconstruction had the file been stored using a volume level implementation of the RAID-X format.

In a distributed, object-based data storage system that includes a plurality of object-based storage devices, and one or more nodes that access virtualized objects from the object-based storage devices, wherein each of the virtualized objects is stored as an aggregation of multiple component objects, each stored on one of the object-based storage devices, and wherein a RAID-X format is implemented at a file level within the plurality of object-based storage devices, a method for reconstruction comprising:

• (a) in response to a failure that renders inaccessible at least one portion of an object-based storage device, retrieving a map for at least one virtualized object having at least one component stored on the at least one portion;
• wherein the map includes a list of storage devices where components of part of the virtualized object reside wherein the part corresponds to less than all parts of the virtualized object;
• (b) using the map to determine whether a number of lost or damaged components associated with the part of the at least one virtualized object exceed a maximum limit beyond which reconstruction of an object stored using the file level implementation of the RAID-X format is not possible;
• and
• (c) if the maximum limit for reconstruction of an object stored using the RAID-X format has not been exceeded, reconstructing the part of the at least one virtualized object;
• wherein the data storage system continues to service data storage and access requests, with reconstruction of the part of the at least one virtualized object being performed as a background operation, after a number of device failures has occurred which would have exceeded the maximum limit of storage device failures for reconstruction had the virtualized object been stored using a volume level implementation of the RAID-X format;
• and
• wherein said at least one virtualized object contains data from one or more logical files, or a portion of a logical file.

In a data storage system that includes a plurality of storage devices, and one or more nodes that access files from the storage devices, wherein a RAID-X format is implemented at a file level within the plurality of storage devices, a method for file reconstruction comprising:

• (a) in response to a failure of one or more of the storage devices, retrieving a map for a file having at least one component stored on the at least one portion;
• wherein the map includes a list of storage devices where components of part of the file reside wherein the part corresponds to less than all parts of the file;
• (b) using the map to determine whether a number of lost or damaged components associated with the part of the file exceed a maximum limit beyond which reconstruction of a file stored using the file level implementation of the RAID-X format is not possible, and identifying one or more lost or damaged parts of files where the maximum limit for reconstruction of the parts of the files stored using the file level implementation of the RAID-X format has been exceeded;
• wherein the data storage system continues to service data storage and access requests, with step (b) being performed as a background operation, after a number of device failures has occurred which would have exceeded the maximum limit of storage device failures for reconstruction had the parts of the files been stored using a volume level implementation of the RAID-X format;
• and
• wherein step (b) further comprises at least one of:
• (i) providing a list of the one or more lost or damaged parts of the files to an administrator in order to facilitate a manual restore from backup of the one or more lost or damaged parts of the files;
• and
• (ii) for each of the one or more lost or damaged parts of the files, automatically retrieving a most recent version of the part of the file from a backup location.

In a distributed, object-based data storage system that includes a plurality of object-based storage devices, and one or more nodes that access virtualized objects from the object-based storage devices, wherein each of the virtualized objects is stored as an aggregation of multiple component objects, each stored on one of the object-based storage devices, and wherein a RAID-X format is implemented at a file level within the plurality of object-based storage devices, a method for reconstruction comprising:

• (a) in response to a failure of one or more of the storage devices, retrieving a map for at least one virtualized object having at least one component stored on the one or more failed storage devices;
• wherein the map includes a list of storage devices where components of part of the virtualized object reside wherein the part corresponds to less than all parts of the virtualized object;
• (b) using the map to determine whether a number of lost or damaged components associated with the part of the at least one virtualized object exceed a maximum limit beyond which reconstruction of a virtualized object stored using the file level implementation of the RAID-X format is not possible, and identifying one or more lost or damaged parts of virtualized objects where the maximum limit for reconstruction of the parts of the virtualized objects stored using the file level implementation of the RAID-X format has been exceeded;
• wherein the data storage system continues to service data storage and access requests, with step (b) being performed as a background operation, after a number of device failures has occurred which would have exceeded the maximum limit of storage device failures for reconstruction had the part of the at least one virtualized object been stored using a volume level implementation of the RAID-X format;
• and
• wherein step (b) further comprises at least one of:
• (i) providing a list of the one or more lost or damaged parts of the virtualized objects to an administrator in order to facilitate a manual restore from backup of the one or more lost or damaged parts of the virtualized objects;
• and
• (ii) for each of the one or more lost or damaged parts of the virtualized objects, automatically retrieving a most recent version of the part of the virtualized object from a backup location;
• wherein said at least one virtualized object contains data from one or more logical files, or a portion of a logical file.