Controlled take over of services by remaining nodes of clustered computing system

A method for taking over services by one or more remaining sub-clusters of a clustered computing system from one or more other sub-clusters of the clustered computing system after the one or more other sub-clusters have been shutdown, said method comprising:

• (a) determining whether one or more computing nodes in a cluster have become one or more non-responsive nodes;
• (b) starting a first timer when said determining (a) determines that one or more of the computing nodes in the cluster have become one or more non-responsive nodes, the first timer having a first duration;
• (c) attempting to determine whether a sub-cluster vote is at least a majority of a total votes available, the sub-cluster vote representing votes for a sub-cluster of one or more computing nodes, the sub-cluster representing a portion of the cluster that remains responsive;
• (d) initiating shutdown of the one or more non-responsive computing nodes of the sub-cluster when said attempting (c) does not determine within the first duration of the first timer that the sub-cluster vote is at least a majority of the total votes available;
• (e) to allow the processors of the sub-cluster to temporarily continue, starting a second timer after the first timer expires when the said attempting (c) has determined within the first duration of the first timer that the sub-cluster vote is at least a majority of the total votes available, the second timer having a second duration such that data corruption is avoided;
• and
• (f) taking over services from the one or more non-responsive nodes by at least one of the remaining computing nodes of the sub-cluster after the second timer expires.

A computer readable medium including computer program code for taking over services by one or more remaining sub-clusters of a clustered computing system from one or more other sub-clusters of the clustered computing system after the one or more other sub-clusters have been shutdown, said computer readable medium comprising:

• computer program code for attempting to determine whether a sub-cluster of the clustered computing system is to remain active such that the processes of the sub-cluster may temporarily continue;
• computer program code for initiating shutdown of the sub-cluster when said computer program code for attempting does not determine within a first predetermined amount of time that the sub-cluster is to remain active;
• computer program code for delaying for a second predetermined amount of time after the first predetermined amount of time expires when said computer program code for attempting determines within the first predetermined amount of time that the sub-cluster is to remain active such that the processes of the sub-cluster may temporarily continue;
• and
• computer program code to allow the processors of the sub-cluster to temporarily continue for taking over services of one or more other sub-clusters of the clustered computing system by one or more remaining sub-clusters after said computer program code for delaying has delayed for the second predetermined amount of time such that data corruption is avoided.

A computer readable medium for taking over services by one or more remaining sub-clusters of a clustered computing system from one or more other sub-clusters of the clustered computing system after the one or more other sub-clusters have been shutdown, said computer readable medium comprising:

• computer program code for determining whether one or more computing nodes in a cluster have become one or more non-responsive nodes;
• computer program code for starting a first timer when said computer program code for determining determines that one or more of the computing nodes in the cluster have become one or more non-responsive nodes, the first timer having a first duration;
• computer program code for attempting to determine whether a sub-cluster vote is at least a majority of a total votes available, the sub-cluster vote representing votes for a sub-cluster of one or more computing nodes, the sub-cluster representing a portion of the cluster that remains responsive;
• computer program code for initiating shutdown of the one or more non-responsive computing nodes of the sub-cluster when said computer program code for attempting does not determine within the first duration of the first timer that the sub-cluster vote is at least a majority of the total votes available;
• computer program code for starting a second timer to allow the processors of the sub-cluster to temporarily continue after the first timer expires when the said computer program code for attempting has determined within the first duration of the first timer that the sub-cluster vote is at least a majority of the total votes available, the second timer having a second duration such that data corruption is avoided;
• and
• computer program code for taking over services from the one or more non-responsive nodes by at least one of the remaining computing nodes of the sub-cluster after the second timer expires.

A clustered computing system, comprising:

• a cluster of computing nodes having at least two computing nodes;
• and
• an integrity protector, comprising:
• a cluster error detector operable to detect a formation of disjoint sub-clusters;
• a cluster shutdown controller operable to:
• attempt to determine whether a sub-cluster of the clustered computing system is to remain active such that the processes of the sub-cluster may temporarily continue;
• initiate shutdown of the sub-cluster when the attempt does not determine within a first predetermined amount of time that the sub-cluster is to remain active;
• and
• to allow the processors of the sub-cluster to temporarily continue, delay for a second predetermined amount of time after the first predetermined amount of time expires when said attempt determines within the first predetermined amount of time that the sub-cluster is to remain active such that data corruption is avoided;
• and
• a takeover controller operable to:
• take over services of one or more other sub-clusters of the clustered computing system by one or more remaining sub-clusters after the delay for the second predetermined amount of time.