Network Fault Detection and Reconfiguration

A computer-readable storage medium storing computer-executable code for detecting communication faults in a parallel system, the code, when executed by a processor, causing the processor to:

• send ping messages by a node of the parallel system to one or more destination nodes, wherein the parallel system comprises a plurality of nodes communicating with each other using a plurality of links, each node comprising a processor;
• wait to receive acknowledgements from each destination node indicating the destination node received a ping message;
• responsive to failure to receive one or more acknowledgement message, detect failure of corresponding one or more ping messages to reach their destination nodes;
• and
• responsive to detecting failure of one or more ping messages to reach their target nodes, identify faulty component in the parallel system, the identifying causing the processor to:
• freeze communications in the parallel system by sending a request to nodes of the system to stop sending and receiving messages except for ping messages;
• send ping messages through different components of the parallel system;
• identify the faulty component based on failure to deliver a ping message through the component;
• and
• unfreeze the parallel system by sending requests to the nodes of the system to restart sending and receiving messages other than ping messages.

A computer-implemented system for detecting communication faults in a parallel system, the system comprising:

• a computer processor;
• and
• a computer-readable storage medium storing computer program modules configured to execute on the computer processor, the computer program modules comprising:
• a communication module configured to:
• send ping messages by a node of the parallel system to one or more destination nodes, wherein the parallel system comprises a plurality of nodes communicating with each other using a plurality of links, each node comprising a processor;
• wait to receive acknowledgements from each destination node indicating the destination node received a ping message;
• a fault detection module configured to:
• responsive to failure to receive one or more acknowledgement message, detect failure of corresponding one or more ping messages to reach their destination nodes;
• and
• responsive to detecting failure of one or more ping messages to reach their target nodes, identify faulty component in the parallel system, the identifying causing the processor to:
• freeze communications in the parallel system by sending a request to nodes of the system to stop sending and receiving messages except for ping messages;
• send ping messages through different components of the parallel system;
• identify the faulty component based on failure to deliver a ping message through the component;
• and
• unfreeze the parallel system by sending requests to the nodes of the system to restart sending and receiving messages other than ping messages.