# US 2008/0189573 A1 · Fault recovery on a massively parallel computer system to handle node failures without ending an executing job > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2008/0189573 A1 - Document ID: US20080189573A1 - Status: Published application - Kind: A1 - Application number: 11/670,803 - Filed: 2007-02-02 - Published: 2008-08-07 - Earliest claimed: 2007-02-02 - Assignees: International Business Machines Corporation - Inventors: David L. Darrington; Patrick Joseph McCarthy; Amanda Peters; Albert Sidelnik - Web page: https://patentgrep.com/patent/US20080189573A1 ## CPC - G06F 11/267: Reconfiguring circuits for testing, e.g. LSSD, partitioning - G06F 1/24: Resetting means - G06F 11/0724: {in a multiprocessor or a multi-core unit (multiprocessors per se G06F15/80)} - G06F 11/0793: {Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40)} - G06F 11/1441: {Resetting or repowering} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method and apparatus for fault recovery of on a parallel computer system from a soft failure without ending an executing job on a partition of nodes. In preferred embodiments a failed hardware recovery mechanism on a service node uses a heartbeat monitor to determine when a node failure occurs. Where possible, the failed node is reset and re-loaded with software without ending the software job being executed by the partition containing the failed node. ## Claims ### Claim 1 A parallel computer system comprising: a plurality of compute nodes each having reset hardware to reset a network hardware portion of the compute node separate from reset hardware to reset a remaining portion of the compute node; a service node for controlling the operation of the compute nodes over a network, the service node including a failed hardware recovery mechanism that detects a failed compute node; and wherein the failed hardware recovery mechanism resets the remaining portion of the failed compute node without resetting the network hardware portion to recover from a fault on the failed compute node. ### Claim 2 (depends on 1) The parallel computer system of claim 1 wherein the plurality of compute nodes further comprise a timer for setting a heart beat flag at predetermined intervals in a memory of the compute node to indicate the compute node is operating properly. ### Claim 3 (depends on 2) The parallel computer system of claim 2 wherein the failed hardware recovery mechanism further comprises a heart beat monitor to monitor the heart beat flag in the compute nodes to detect a failed compute node of the plurality of nodes by the lack of a set heart beat flag. ### Claim 4 (depends on 3) The parallel computer system of claim 3 wherein the fault on the failed compute node is detected by the heart beat monitor. ### Claim 5 (depends on 3) The parallel computer system of claim 3 wherein the heart beat flag is stored in static memory on the compute node and the failed hardware recovery mechanism reads the static memory over an ethernet network that accesses the static memory through a JTAG interface on the compute node. ### Claim 6 (depends on 5) The parallel computer system of claim 5 wherein the parallel computer system is a massively parallel computer system with a large number of compute nodes housed in a number of closely arranged computer racks. ### Claim 7 (depends on 1) The parallel computer system of claim 1 wherein the remaining portion of the compute node is a DDR memory controller of an ASIC processor chip. ### Claim 8 (depends on 1) The parallel computer system of claim 1 wherein the parallel computer system is a massively parallel computer system with a large number of compute nodes housed in a number of closely arranged computer racks. ### Claim 9 A computer implemented method for operating a parallel computer system having a plurality of compute nodes connected to a service node by a control system network, the method comprising the steps of: a) each node providing a heart beat; b) monitoring the heart beat of each compute node in the service node of the computer system; and c) attempting to recover from a fault in the compute node indicated by a lack of a heart beat in the compute node without aborting an application running on a partition of nodes containing the compute node with the fault. ### Claim 10 (depends on 9) The computer implemented method of claim 9 wherein the step of monitoring the compute node comprises the steps of: d) the compute node receiving a heart beat time from the service node; e) setting a timer with the heart beat time; and f) detecting the elapsed heart beat time of the timer and setting a heart beat flag in a memory of the compute node. ### Claim 11 (depends on 9) The computer implemented method of claim 9 wherein the step of attempting to recover from a fault in the compute node comprises the steps of: g) attempting to load diagnostic code into the compute node; and h) if the load is not successful then resetting a portion of the compute node, including all sections of the compute node except a network hardware section, resetting a memory controller in the compute node and loading a system kernel in the compute node. ### Claim 12 (depends on 11) The computer implemented method of claim 11 further comprising the steps of: i) if the load is successful then executing the diagnostic code to check a memory controller for proper operation; and j) if the memory controller is operating properly, then loading code in the compute node to reset the memory controller, resetting the memory controller and loading the compute node with a system kernel. ### Claim 13 (depends on 12) The computer implemented method of claim 12 wherein memory controller is a DDR memory controller of an ASIC processor chip. ### Claim 14 (depends on 9) The computer implemented method of claim 9 wherein the computer system is a massively parallel computer system housed in a number of closely arranged computer racks. ### Claim 15 A computer-readable program product comprising: a failed hardware recovery mechanism that monitors a plurality of compute nodes in a parallel computer system to determine if a compute node fails, and wherein the failed hardware recovery mechanism resets a remaining portion of a failed compute node without resetting a network hardware portion of the compute node to recover from a fault on the failed compute node; and recordable media bearing the failed hardware recovery mechanism. ### Claim 16 (depends on 15) The program product of claim 15 wherein the plurality of compute nodes further comprise a timer for setting a heart beat flag at predetermined intervals in a memory of the compute node to indicate the compute node is operating properly. ### Claim 17 (depends on 16) The program product of claim 16 wherein the failed hardware recovery mechanism further comprises a heart beat monitor to monitor a heart beat in the compute nodes to detect a failed compute node of the plurality of nodes by the lack of the set heart beat flag. ### Claim 18 (depends on 17) The program product of claim 17 wherein the heart beat flag is stored in static memory on the compute node and the service node reads the static memory over a ethernet network that accesses the static memory through a JTAG interface on the compute node. ### Claim 19 (depends on 18) The program product of claim 18 wherein the parallel computer system is a massively parallel computer system with a large number of compute nodes housed in a number of closely arranged computer racks. ### Claim 20 (depends on 15) The program product of claim 15 wherein the parallel computer system is a massively parallel computer system with a large number of compute nodes housed in a number of closely arranged computer racks.