# US 2015/0149835 A1 · Managing Faults in a High Availability System

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0149835 A1
- Document ID: US20150149835A1
- Status: Published application
- Kind: A1
- Application number: 14/089,852
- Filed: 2013-11-26
- Published: 2015-05-28
- Earliest claimed: 2013-11-26
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Arunachalam Jayaraman
- Web page: https://patentgrep.com/patent/US20150149835A1

## CPC

- G06F 11/0751: {Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06)}
- G06F 11/008: {Reliability or availability analysis}
- G06F 11/0706: {the processing taking place on a specific hardware platform or in a specific software environment}
- G06F 11/0769: {Readable error formats, e.g. cross-platform generic formats, human understandable formats}
- G06F 11/079: {Root cause analysis, i.e. error or fault diagnosis (in a hardware test environment G06F11/22; in a software test environment G06F11/36)}
- G06F 11/2023: {Failover techniques}
- G06F 11/2094: {Redundant storage or storage space (G06F11/2056 takes precedence)}
- G06F 11/3419: {by assessing time}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An approach is provided for managing a failure of a critical high availability (HA) component in a HA system. Critical HA components are identified. Categories are assigned to the identified components and weights are assigned to the categories. A current value indicating a performance of a component included in the identified components is obtained by periodically monitoring the components. A reference value for the performance of the component is received. A deviation between the current value and the reference value is determined. Based on the deviation, the component is determined to have failed. Based in part on the failed component, the categories, and the weights, a health index is determined in real-time. The health index indicates in part how much the component having failed affects a measure of health of the HA system.

## Claims

### Claim 1

A method of managing a failure of a critical high availability (HA) component, the method comprising the steps of:
a computer identifying a plurality of critical HA components of a HA system;
the computer receiving and assigning categories to the plurality of critical HA components;
the computer receiving and assigning weights to the categories;
the computer obtaining a current value indicating a performance of a critical HA component included in the plurality of critical HA components, the current value obtained by periodically monitoring the plurality of critical HA components;
the computer receiving a reference value for the performance of the critical HA component;
the computer determining a deviation between the current value and the reference value;
based on the deviation, the computer determining the critical HA component has failed; and
based in part on the failed critical HA component, the categories, and the weights, the computer determining a health index in real-time, the health index indicating in part how much the critical HA component having failed affects a measure of health of the HA system.

### Claim 2 (depends on 1)

The method of claim 1, further comprising the steps of:
the computer determining n counts C1 . . . Cn, wherein Ci in C1 . . . Cn is a count of how many of the critical HA components are assigned to an i-th category of the categories assigned to the critical HA components; and
the computer determining n counts FC1 . . . FCn, wherein FCi in FC1 . . . FCn is a count of how many of the critical HA components that are assigned in the i-th category have a failed status,
wherein the categories consist of n categories, wherein the weights consist of n weights W1 . . . Wn, wherein n is an integer greater than one, wherein the step of determining the health index includes the step of the computer determining: {{{C1×W1)+ . . . +(Cn×Wn)}−{(FC1×W1)+ . . . +(FCn×Wn)}}/{C1×W1)+ . . . +(Cn×Wn}}*100%.

### Claim 3 (depends on 1)

The method of claim 1, further comprising the steps of:
the computer determining the health index is not equal to 100%;
based on the health index not being equal to 100%, the computer receiving indexes corresponding to respective critical HA components, each index indicating whether the respective critical HA component is an immediately recoverable component;
based on the received index corresponding to the critical HA component that has failed, the computer determining whether the critical HA component that has failed is an immediately recoverable component, in response to a determination that the critical HA component that has failed is the immediately recoverable component, the computer performing branch based decision making to automatically recover the critical HA component, and in response to a determination that the critical HA component that has failed is not the immediately recoverable component, the computer generating a ticket indicating a need for human intervention to recover the critical HA component.

### Claim 4 (depends on 3)

The method of claim 3, wherein the step of determining whether the critical HA component that has failed is the immediately recoverable component includes determining whether the critical HA component is a hardware component or a software component, the critical HA component being the hardware component indicating the critical HA component is not the immediately recoverable component, and the critical HA component being the software component indicating the critical HA component is the immediately recoverable component.

### Claim 5 (depends on 3)

The method of claim 3, wherein the step of performing the branch based decision making includes the steps of:
the computer determining whether the critical HA component is assigned to a leaf category included in the categories, wherein the categories consist of the leaf category, a stem category, and a root category;
in response to a determination that the critical HA component is assigned to the leaf category, the computer automatically recovering the critical HA component by invoking a self-healing script for the leaf category;
in response to a determination that the critical HA component is not assigned to the leaf category, the computer determining whether the critical HA component is assigned to the stem category;
in response to a determination that the critical HA component is assigned to the stem category, the computer automatically recovering the critical HA component by invoking a self-healing script for the stem category;
in response to a determination that the critical HA component is not assigned to the stem category, the computer determining whether the critical HA component is assigned to the root category;
in response to a determination that the critical HA component is assigned to the root category, the computer automatically recovering the critical HA component by invoking a self-healing script for the root category; and
in response to a determination that the critical HA component is not assigned to the root category, the computer generating a ticket indicating a self-healing of the critical HA component has failed.

### Claim 6 (depends on 5)

The method of claim 5, further comprising the steps of:
subsequent to recovering the critical HA component by invoking the self-healing script for the leaf, stem or root category, the computer identifying a peer index of the critical HA component;
based on the peer index, the computer determining a peer of the critical HA component, the peer index indicating the peer and the critical HA component have substantially similar functions in a subsystem;
the computer determining whether a weight assigned to the peer indicates the peer is in the stem, leaf, or root category;
in response to a determination that the weight assigned to the peer indicates the peer is in the stem category,
the computer traversing up a branch from the peer to one or more critical HA components in the leaf category, performing the branch based decision making on the one or more critical HA components to determine if any of the one or more critical HA components have failed, and performing a self-healing to recover any of the one or more critical HA components that have failed; and
the computer traversing down the branch from the peer to a critical HA component in the root category, performing the branch based decision making to determine if the critical HA component in the root category has failed, and performing a self-healing to recover the critical HA component in the root category that has failed;
in response to a determination that the weight assigned to the peer indicates the peer is in the leaf category, the computer traversing down the branch from the peer to a critical HA component in the stem category, performing the branch based decision making to determine if the critical HA component in the stem category has failed, and performing a self-healing to recover the critical HA component in stem category that has failed; and
in response to a determination that the weight assigned to the peer indicates the peer is in the root category, the computer traversing up the branch from the peer to one or more critical HA components in the stem category, performing the branch based decision making to determine if any of the one or more critical HA components in the stem category have failed, and performing a self-healing to recover any of the one or more critical HA components in the stem category that have failed.

### Claim 7 (depends on 1)

The method of claim 1, further comprising the step of:
providing at least one support service for at least one of creating, integrating, hosting, maintaining, and deploying computer-readable program code in the computer, the program code being executed by a processor of the computer to implement the steps of identifying the plurality of critical HA components, receiving and assigning the categories to the plurality of critical HA components, receiving and assigning the weights to the categories, obtaining the current value indicating the performance of the critical HA component, receiving the reference value for the performance of the critical HA component, determining the deviation between the current value and the reference value, determining the critical HA component has failed, and determining the health index in real-time.

### Claim 8 (depends on 1)

The computer of claim 1, one or more processors of the computer, one or more computer-readable memories, one or more computer-readable storage devices, other computer hardware, and program instructions which are stored on the one or more storage devices for execution by the one or more processors via the one or more memories and when executed by the one or more processors with the other computer hardware perform the method of claim 1.

### Claim 9

A computer program product comprising:
a computer-readable, tangible storage device; and
a computer-readable program code stored in the computer-readable, tangible storage device, the computer-readable program code containing instructions that are executed by a central processing unit (CPU) of a computer system to implement a method of managing a failure of a critical high availability (HA) component, the method comprising the steps of:
the computer system identifying a plurality of critical HA components of a HA system;
the computer system receiving and assigning categories to the plurality of critical HA components;
the computer system receiving and assigning weights to the categories;
the computer system obtaining a current value indicating a performance of a critical HA component included in the plurality of critical HA components, the current value obtained by periodically monitoring the plurality of critical HA components;
the computer system receiving a reference value for the performance of the critical HA component;
the computer system determining a deviation between the current value and the reference value;
based on the deviation, the computer system determining the critical HA component has failed; and
based in part on the failed critical HA component, the categories, and the weights, the computer system determining a health index in real-time, the health index indicating in part how much the critical HA component having failed affects a measure of health of the HA system.

### Claim 10 (depends on 9)

The program product of claim 9, wherein the method further comprises the steps of:
the computer system determining n counts C1 . . . Cn, wherein Ci in C1 . . . Cn is a count of how many of the critical HA components are assigned to an i-th category of the categories assigned to the critical HA components; and
the computer system determining n counts FC1 . . . FCn, wherein FCi in FC1 . . . FCn is a count of how many of the critical HA components that are assigned in the i-th category have a failed status,
wherein the categories consist of n categories, wherein the weights consist of n weights W1 . . . Wn, wherein n is an integer greater than one, wherein the step of determining the health index includes the step of the computer system determining: {{{C1×W1)+ . . . +(Cn×Wn)}−{(FC1×W1)+ . . . +(FCn×Wn)}}/{C1×W1)+ . . . +(Cn×Wn}}*100%.

### Claim 11 (depends on 9)

The program product of claim 9, wherein the method further comprises the steps of:
the computer system determining the health index is not equal to 100%;
based on the health index not being equal to 100%, the computer system receiving indexes corresponding to respective critical HA components, each index indicating whether the respective critical HA component is an immediately recoverable component;
based on the received index corresponding to the critical HA component that has failed, the computer system determining whether the critical HA component that has failed is an immediately recoverable component, in response to a determination that the critical HA component that has failed is the immediately recoverable component, the computer system performing branch based decision making to automatically recover the critical HA component, and in response to a determination that the critical HA component that has failed is not the immediately recoverable component, the computer system generating a ticket indicating a need for human intervention to recover the critical HA component.

### Claim 12 (depends on 11)

The program product of claim 11, wherein the step of determining whether the critical HA component that has failed is the immediately recoverable component includes determining whether the critical HA component is a hardware component or a software component, the critical HA component being the hardware component indicating the critical HA component is not the immediately recoverable component, and the critical HA component being the software component indicating the critical HA component is the immediately recoverable component.

### Claim 13 (depends on 11)

The program product of claim 11, wherein the step of performing the branch based decision making includes the steps of:
the computer system determining whether the critical HA component is assigned to a leaf category included in the categories, wherein the categories consist of the leaf category, a stem category, and a root category;
in response to a determination that the critical HA component is assigned to the leaf category, the computer system automatically recovering the critical HA component by invoking a self-healing script for the leaf category;
in response to a determination that the critical HA component is not assigned to the leaf category, the computer system determining whether the critical HA component is assigned to the stem category;
in response to a determination that the critical HA component is assigned to the stem category, the computer system automatically recovering the critical HA component by invoking a self-healing script for the stem category;
in response to a determination that the critical HA component is not assigned to the stem category, the computer system determining whether the critical HA component is assigned to the root category;
in response to a determination that the critical HA component is assigned to the root category, the computer system automatically recovering the critical HA component by invoking a self-healing script for the root category; and
in response to a determination that the critical HA component is not assigned to the root category, the computer system generating a ticket indicating a self-healing of the critical HA component has failed.

### Claim 14 (depends on 13)

The program product of claim 13, wherein the method further comprises the steps of:
subsequent to recovering the critical HA component by invoking the self-healing script for the leaf, stem or root category, the computer system identifying a peer index of the critical HA component;
based on the peer index, the computer system determining a peer of the critical HA component, the peer index indicating the peer and the critical HA component have substantially similar functions in a subsystem;
the computer system determining whether a weight assigned to the peer indicates the peer is in the stem, leaf, or root category;
in response to a determination that the weight assigned to the peer indicates the peer is in the stem category,
the computer system traversing up a branch from the peer to one or more critical HA components in the leaf category, performing the branch based decision making on the one or more critical HA components to determine if any of the one or more critical HA components have failed, and performing a self-healing to recover any of the one or more critical HA components that have failed; and
the computer system traversing down the branch from the peer to a critical HA component in the root category, performing the branch based decision making to determine if the critical HA component in the root category has failed, and performing a self-healing to recover the critical HA component in the root category that has failed;
in response to a determination that the weight assigned to the peer indicates the peer is in the leaf category, the computer system traversing down the branch from the peer to a critical HA component in the stem category, performing the branch based decision making to determine if the critical HA component in the stem category has failed, and performing a self-healing to recover the critical HA component in stem category that has failed; and
in response to a determination that the weight assigned to the peer indicates the peer is in the root category, the computer system traversing up the branch from the peer to one or more critical HA components in the stem category, performing the branch based decision making to determine if any of the one or more critical HA components in the stem category have failed, and performing a self-healing to recover any of the one or more critical HA components in the stem category that have failed.

### Claim 15

A method of predicting a reoccurrence of a failure of a critical high availability (HA) component, the method comprising the steps of:
a computer determining a real-time failover is happening based on a failure of a HA system;
the computer determining an actual amount of time taken by an event occurring during the failover;
the computer receiving a reference amount of time that the event occurring during the failover is expected to take;
the computer determining the actual amount of time is not equal to the reference amount of time within a predefined tolerance;
based on the actual amount of time being not equal to the reference amount of time, the computer predicting the failure will reoccur unless a fault is repaired;
the computer identifying critical HA components participating in the event, the critical HA components included in the HA system; and
the computer determining a critical HA component included in the identified critical HA components has failed based on the fault and invoking self-healing to repair the fault in the critical HA component by performing a branch based decision making process on the identified critical HA components.

### Claim 16 (depends on 15)

The method of claim 15, wherein the step of determining the actual amount of time taken by the event occurring during the failover includes:
the computer monitoring in real time a change in a state of the critical HA component; and
the computer recording an actual amount of time taken by the change in the state of the critical HA component.

### Claim 17 (depends on 16)

The method of claim 16, wherein the step of recording the actual amount of time taken by the change includes recording an amount of time for the critical HA component to change the state from online to offline or from offline to online.

### Claim 18 (depends on 16)

The method of claim 16, wherein the step of identifying the critical HA components participating in the event includes identifying first, second and third critical HA components in a branch including the first critical HA component in a root category, the second critical HA component in a stem category, and the third critical HA component in a leaf category, and
wherein the step of performing the branch based decision making process includes performing the branch based decision making process on the first, second and third critical HA components.

### Claim 19 (depends on 15)

The method of claim 15, further comprising the step of:
providing at least one support service for at least one of creating, integrating, hosting, maintaining, and deploying computer-readable program code in the computer, the program code being executed by a processor of the computer to implement the steps of determining the real-time failover is happening, determining the actual amount of time, receiving the reference amount of time, determining the actual amount of time is not equal to the reference amount of time, the computer predicting the failure will reoccur, identifying critical HA components participating in the event, and determining the critical HA component has failed and invoking self-healing to repair the fault in the critical HA component by performing the branch based decision making process on the identified critical HA components.

### Claim 20 (depends on 15)

The computer of claim 15, one or more processors of the computer, one or more computer-readable memories, one or more computer-readable storage devices, other computer hardware, and program instructions which are stored on the one or more storage devices for execution by the one or more processors via the one or more memories and when executed by the one or more processors with the other computer hardware perform the method of claim 15.
