# US 9,519,545 B2 · Storage drive remediation in a raid system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,519,545 B2 - Document ID: US9519545B2 - Status: Granted - Kind: B2 - Application number: 14/538,703 - Filed: 2014-11-11 - Published: 2016-05-12 - Granted: 2016-12-13 - Earliest claimed: 2014-11-11 - Assignees: Facebook, Inc. - Applicants: Facebook, Inc. - Inventors: Mateusz Marek Niewczas - Web page: https://patentgrep.com/patent/US9519545B2 ## CPC - G06F 11/1441: {Resetting or repowering} - G06F 3/06: Digital input from, or digital output to, record carriers {, e.g. RAID, emulated record carriers or networked record carriers} - G06F 11/0727: {in a storage system, e.g. in a DASD or network based storage system (drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)} - G06F 11/1092: {Rebuilding, e.g. when physically replacing a failing disk} - G06F 11/1417: {Boot up procedures} - G06F 11/0709: {in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems} - G06F 11/0793: {Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40)} - G06F 11/108: {Parity data distribution in semiconductor storages, e.g. in SSD} - G06F 11/1088: {Reconstruction on already foreseen single or plurality of spare disks} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Techniques, systems, and devices are disclosed for remediating a failed drive in a set of drives, such as a RAID system, without having to physically replace the failed drive. After receiving a signal of an error indicating a specific physical portion on a storage drive in the set of storage drives has caused the drive to fail, the system can unmount the drive from the filesystem while other drives continue to operate. Next, the system can identify one or more files in the filesystem that have associations with the specific physical portion on the failed drive. Next, the system can remount the drive onto the filesystem and subsequently delete the identified files from the filesystem. The system can then perform a direct I/O write to the specific physical portion on the failed drive to force reallocation of the specific physical portion to a different area on the failed drive. The system can also power-cycle the drive before this remediation, e.g., to determine if this remediation can be avoided. ## Claims ### Claim 1 A method performed by a computing device having a processor and memory for remediating a failed storage drive in a redundant array of independent disks (RAID) system comprising a set of storage drives and configured for use with a filesystem, comprising: receiving, by the computing device, a signal of an error indicating a specific physical portion on a storage drive in the set of storage drives has caused the storage drive to fail; unmounting, by the computing device, the storage drive from the filesystem while other storage drives in the set of storage drives continue to operate; identifying, by the computing device, one or more files in the filesystem that have associations with the specific physical portion on the storage drive; responsive to identifying the one or more files, remounting, by the computing device, the storage drive onto the filesystem; deleting, by the computing device, the identified one or more files from the filesystem; and performing, by the computing device, a direct I/O write to the specific physical portion on the storage drive to force reallocation of the specific physical portion to a different area on the storage drive. ### Claim 2 (depends on 1) The method of claim 1, wherein prior to unmounting the storage drive from the filesystem, the method further comprises: performing a power-cycling remediation on the storage drive; determining whether power-cycling the storage drive has fixed the error in the storage drive; and if so, exiting the storage drive remediation process. ### Claim 3 (depends on 2) The method of claim 2, wherein performing the power-cycling remediation on the storage drive includes: attempting to capture first storage drive information of the storage drive before power-cycling the storage drive; power-cycling the storage drive; and attempting to capture second storage drive information of the storage drive after the power cycling of the storage drive. ### Claim 4 (depends on 3) The method of claim 3, wherein determining whether power-cycling the storage drive has fixed the error in the storage drive includes: comparing the captured first storage drive information with the captured second storage drive information; and analyzing the differences between the first storage drive information and the second storage drive information to determine whether the error in the storage drive has been fixed. ### Claim 5 (depends on 3) The method of claim 3, wherein the storage drive information includes self-monitoring, analysis, and reporting technology (SMART) data and metadata. ### Claim 6 (depends on 1) The method of claim 1, wherein prior to unmounting the storage drive from the filesystem, the method further comprises: attempting to capture first storage drive information of the storage drive before power-cycling the storage drive; power-cycling the storage drive; attempting to capture second storage drive information of the storage drive after the power cycling of the storage drive; and if both attempts to capture the first storage drive information and the second storage drive information have failed, determining that the storage drive is not remediable and stopping the remediation operation. ### Claim 7 (depends on 6) The method of claim 6, wherein after determining that the storage drive is not remediable, the method further comprises generating a decision to replace the storage drive. ### Claim 8 (depends on 1) The method of claim 1, wherein the RAID system operates in conjunction with a Hadoop application, and wherein unmounting the storage drive from the filesystem includes: halting the Hadoop application; unmounting the storage drive from the filesystem; provisioning nodes for the Hadoop application; and restarting the Hadoop application. ### Claim 9 (depends on 8) The method of claim 8, wherein the Hadoop application includes a Hadoop distributed file system (HDFS). ### Claim 10 (depends on 9) The method of claim 9, wherein the method for remediating the failed drive in the RAID system is controlled by the HDFS. ### Claim 11 (depends on 1) The method of claim 1, wherein identifying the one or more files in the filesystem that have associations with the specific physical portion on the storage drive includes: identifying physical sectors within the specific physical portion on the storage drive; and for each of the identified physical sectors, mapping a given physical sector to a first logical block address (LBA) recognized by the RAID system; mapping the first LBA to a second LBA referenced by the filesystem; and scanning the filesystem to identify a file that contains the second LBA. ### Claim 12 (depends on 11) The method of claim 11, wherein scanning the filesystem includes scanning an inode map of the filesystem to identify an inode based on the second LBA. ### Claim 13 (depends on 12) The method of claim 12, wherein deleting a given file in the identified one or more files in the filesystem includes deleting a corresponding inode in the filesystem representing the given file. ### Claim 14 (depends on 12) The method of claim 12, wherein identifying the one or more files in the filesystem that have associations with the specific physical portion on the storage drive further includes identifying files that have associations to the identified inode, including hidden files, symbolic links, or other pointers to the identified inode. ### Claim 15 (depends on 1) The method of claim 1, wherein the filesystem includes an XFS. ### Claim 16 (depends on 1) The method of claim 1, wherein the RAID system includes a RAID 0. ### Claim 17 A non-transitory computer-readable storage medium storing instructions for remediating a failed storage drive in a redundant array of independent disks (RAID) system comprising a set of storage drives and configured for use with a filesystem, comprising: instructions for receiving a signal of an error indicating a specific physical portion on a storage drive in the set of storage drives has caused the storage drive to fail; instructions for unmounting the storage drive from the filesystem while other storage drives in the set of storage drives continue to operate; instructions for identifying one or more files in the filesystem that have associations with the specific physical portion on the storage drive; instructions for remounting the storage drive onto the filesystem; instructions for deleting the identified one or more files in the filesystem; and instructions for performing a direct I/O write to the specific physical portion on the storage drive to force reallocation of the specific physical portion to a different area on the storage drive. ### Claim 18 (depends on 17) The non-transitory computer-readable storage medium of claim 17, wherein identifying the one or more files in the filesystem that have associations with the specific physical portion on the storage drive includes: identifying physical sectors within the specific physical portion on the storage drive; and for each of the identified physical sectors, mapping a given physical sector to a first logical block address (LBA) recognized by the RAID system; mapping the first LBA to a second LBA referenced by the filesystem; and scanning the filesystem to identify a file that contains the second LBA. ### Claim 19 A storage array system, comprising: a computer including: a processor; and a memory storing a filesystem; and a redundant array of independent disks (RAID) system coupled to the computer and configured for use with the filesystem, wherein the RAID system comprises: a set of storage drives; and a RAID controller coupled between the set of storage drives and the computer and configured to manage operations of each of the set of storage drives, wherein the processor in collaboration with the memory is operable to remediate a failed storage drive in the set of storage drives by: receiving a signal of an error indicating a specific physical portion on a storage drive in the set of storage drives has caused the storage drive to fail; unmounting the storage drive from the filesystem while other storage drives in the set of storage drives continue to operate; identifying one or more files in the filesystem that have associations with the specific physical portion on the storage drive; remounting the storage drive onto the filesystem; deleting the identified one or more files in the filesystem; and performing a direct I/O write to the specific physical portion on the storage drive to force reallocation of the specific physical portion to a different area on the storage drive. ### Claim 20 (depends on 19) The storage array system of claim 19, wherein the set of storage drives includes one of: a set of hard disk drives (HDDs); a set of solid state drives (SSDs); a set of hybrid drives of HDDs and SSDs; a set of solid state hybrid drives (SSHDs); a set of optical drives; and a combination of the above.