Fault recovery on a massively parallel computer system to handle node failures without ending an executing job
AssigneeInternational Business Machines Corporation
InventorsDavid L. Darrington, Patrick Joseph McCarthy, Amanda Peters, Albert Sidelnik
Life of this patent
Estimating the term…
Abstract
A method and apparatus for fault recovery of on a parallel computer system from a soft failure without ending an executing job on a partition of nodes. In preferred embodiments a failed hardware recovery mechanism on a service node uses a heartbeat monitor to determine when a node failure occurs. Where possible, the failed node is reset and re-loaded with software without ending the software job being executed by the partition containing the failed node.
Claims20
A parallel computer system comprising:
- a plurality of compute nodes each having reset hardware to reset a network hardware portion of the compute node separate from reset hardware to reset a remaining portion of the compute node;
- a service node for controlling the operation of the compute nodes over a network, the service node including a failed hardware recovery mechanism that detects a failed compute node;
- and wherein the failed hardware recovery mechanism resets the remaining portion of the failed compute node without resetting the network hardware portion to recover from a fault on the failed compute node.
A computer implemented method for operating a parallel computer system having a plurality of compute nodes connected to a service node by a control system network, the method comprising the steps of:
- a) each node providing a heart beat;
- b) monitoring the heart beat of each compute node in the service node of the computer system;
- and
- c) attempting to recover from a fault in the compute node indicated by a lack of a heart beat in the compute node without aborting an application running on a partition of nodes containing the compute node with the fault.
A computer-readable program product comprising:
- a failed hardware recovery mechanism that monitors a plurality of compute nodes in a parallel computer system to determine if a compute node fails, and wherein the failed hardware recovery mechanism resets a remaining portion of a failed compute node without resetting a network hardware portion of the compute node to recover from a fault on the failed compute node;
- and
- recordable media bearing the failed hardware recovery mechanism.
Dependencies are inferred from the claim text.