# US 8,751,866 B2 · Autonomic fault isolation in a highly interconnected system > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 8,751,866 B2 - Document ID: US8751866B2 - Status: Granted - Kind: B2 - Application number: 11/536,360 - Filed: 2006-09-28 - Published: 2008-06-26 - Granted: 2014-06-10 - Earliest claimed: 2006-09-28 - Assignees: International Business Machines Corporation - Applicants: Thomas Michael Gooding; Brian Paul Wallenfelt - Inventors: Thomas Michael Gooding; Brian Paul Wallenfelt - Web page: https://patentgrep.com/patent/US8751866B2 ## CPC - G06F 11/079: {Root cause analysis, i.e. error or fault diagnosis (in a hardware test environment G06F11/22; in a software test environment G06F11/36)} - G06F 11/0724: {in a multiprocessor or a multi-core unit (multiprocessors per se G06F15/80)} - G06F 11/0727: {in a storage system, e.g. in a DASD or network based storage system (drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method and apparatus provides fault isolation in a in a highly interconnected system. A fault isolator uses a virtual topology of the system to autonomically isolate the fault. Failing nodes interconnects are identified and then used to traverse a virtual topology to determine the most suspect components as the cause of the reported failures. A failure indication is propagated to each component connected to a failing node in the virtual topology until the next node in the virtual topology is encountered. In preferred embodiments, the failure indications for each component are tallied together as the topology is traversed for each failing node and/or component. A total count of the number of nodes connected to each component is determined for a denominator. A weighting of the likelihood of fault is then determined by dividing the failure indications by the total count. ## Claims ### Claim 1 A highly interconnected computer system comprising: a plurality of interconnected compute nodes; and a service node that receives failure indications of system components that have a failure, including from the compute nodes in the highly interconnected system comprising: a virtual topology of the highly interconnected computer system; a failure indication count for each system component in the virtual topology; and a fault isolator to autonomically isolate faults in the system by traversing the virtual topology and propagating failure indications from the system components, wherein the fault isolator propagates each failure indication by incrementing the failure indication count of each component in the virtual topology connected to the system component with the failure. ### Claim 2 (depends on 1) The highly interconnected computer system of claim 1 wherein the fault isolator determines a weighting of the likelihood of a component causing the fault by comparing a total number of failure indications propagated to each component with a total count of the components that are capable of reporting a failure connected to each component. ### Claim 3 (depends on 2) The highly interconnected computer system of claim 2 wherein the weighting further includes the hopcount of the component. ### Claim 4 (depends on 2) The highly interconnected computer system of claim 2 wherein the weighting is displayed to the user. ### Claim 5 (depends on 1) The highly interconnected computer system of claim 1 wherein the highly interconnected computer system is a massively parallel computer system. ### Claim 6 (depends on 1) The highly interconnected computer system of claim 1 wherein the fault isolator traverses each component in the virtual topology until another node is encountered. ### Claim 7 A computer implemented method for fault isolation on a highly interconnected computer system comprising the steps of: a) preparing a list of system components of the highly interconnected computer system with a failure; b) traversing a virtual topology of the highly interconnected computer system to propagate a failure indication for each system component on the list, wherein the failure indication is propagated by incrementing a failure indication count of each component in the virtual topology connected to the system component with the failure; and c) weighting each component in the virtual topology based on the likelihood that component is the cause of a fault. ### Claim 8 (depends on 7) The computer implemented method of claim 7 wherein the step of traversing the virtual topology further comprises the steps of: 1) filtering links that are not in failure; 2) traversing the unfiltered links and adding neighboring components to the end of a component search list; 3) selecting the next component in the component search list to search until all the components in the component search list have been searched; 4) incrementing a fail count of each component connected to the selected component that has not been searched; and 5) if the selected component is not a node, then returning to step 1 to traverse links of the selected component to further search the link. ### Claim 9 (depends on 7) The computer implemented method of claim 7 wherein the step of weighting compares a total number of failure indications for the component with a total count of the components that are capable of reporting a failure connected to the component. ### Claim 10 (depends on 9) The computer implemented method of claim 9 wherein the weighting further includes the hopcount of the component. ### Claim 11 (depends on 9) The computer implemented method of claim 9 wherein the weighting is displayed to the user. ### Claim 12 (depends on 7) The computer implemented method of claim 7 wherein the highly interconnected computer system is a massively parallel computer system. ### Claim 13 (depends on 7) The computer implemented method of claim 7 wherein the fault isolator traverses the virtual topology until another node is encountered. ### Claim 14 A computer-readable program product comprising: a fault isolator that autonomically isolates faults in a highly interconnect computer system by traversing a virtual topology and propagating failure indications from components of the highly interconnected computer system that have a failure, wherein each failure indication is propagated by incrementing a failure indication count of each component in the virtual topology connected to the system component with the failure; and recordable media bearing the inbound packet filter mechanism. ### Claim 15 (depends on 14) The program product of claim 14 wherein the fault isolator determines a weighting of the likelihood of a component causing the fault by comparing a total number of failure indications for the component with a total count of the components that are capable of reporting a failure connected to the component. ### Claim 16 (depends on 15) The program product of claim 15 wherein the weighting further includes the hopcount of the component. ### Claim 17 (depends on 15) The program product of claim 15 wherein the weighting is displayed to the user. ### Claim 18 (depends on 14) The program product of claim 14 wherein the highly interconnected computer system is a massively parallel computer system. ### Claim 19 (depends on 14) The program product of claim 14 wherein the fault isolator traverses the virtual topology until another node is encountered.