Methods and apparatus for detecting, eliminating and/or mitigating split brain occurrences in high availability systems

A high availability system comprising:

• a cluster of processing nodes, said cluster of processing nodes including a standby processing node and an active processing node;
• and
• wherein the standby processing node includes:
• (i) memory,
• (ii) a first health check probing Internet Protocol (IP) communications interface, and
• (iii) a first processor, the first processor controlling the standby processing node to perform the following operations:
• determining that a failure condition exists;
• in response to determining, by the standby processing node, that a failure condition exists, making a determination, by the standby processing node, whether switching from a standby mode of operation to an active mode of operation will result in a split brain condition for the high availability system;
• and
• when the determination is that switching from a standby mode of operation to an active mode of operation will result in a split brain condition for the high availability system, refraining, by the standby processing node, from switching from a standby mode of operation to an active mode of operation;
• and
• wherein said operation of making a determination, by the standby processing node, whether switching from a standby mode of operation to an active mode of operation will result in a split brain condition for the high availability system includes:
• determining, by the standby processing node, whether or not the active processing node has failed;
• and
• when the determination is that the active processing node has not failed making the determination that switching by the standby processing node from a standby mode of operation to an active mode of operation, will result in a split brain condition for the high availability system;
• and
• wherein said determining, by the standby processing node, whether or not the active processing node has failed includes:
• retrieving, by the standby processing node, health check probe information for the active processing node from memory of the standby processing node;
• determining whether the retrieved health check probe information is stale;
• and
• when said health check probe information is determined not to be stale, determining whether or not the active processing node has failed based on the retrieved health check probe information;
• and
• when said health check probe information is determined to be stale sending, by the standby processing node, a health check probe message from the first health check probing IP communications interface of the standby processing node to a second health check probing IP communications interface of the active processing node.

A non-transitory computer readable medium including a first set of computer executable instructions which when executed by a processor of a standby processing node of a high availability system including a cluster of processing nodes cause the standby processing node to:

• determine that a failure condition exists;
• in response to determining by the standby processing node that a failure condition exists, making a determination, by the standby processing node, whether switching from a standby mode of operation to an active mode of operation, will result in a split brain condition for the high availability system;
• and
• when the determination is that switching from a standby mode of operation to an active mode of operation will result in a split brain condition for the high availability system, refraining, by the standby processing node, from switching from a standby mode of operation to an active mode of operation;
• wherein said making a determination, by the standby processing node, whether switching from a standby mode of operation to an active mode of operation will result in a split brain condition for the high availability system includes:
• determining, by the standby processing node, whether or not an active processing node of the cluster of processing nodes has failed;
• and
• when the determination is that the active processing node has not failed, making the determination that switching, by the standby processing node, from a standby mode of operation to an active mode of operation will result in a split brain condition for the high availability system;
• and
• wherein said determining, by the standby processing node, whether or not the active processing node of the cluster of processing nodes has failed includes:
• retrieving, by the standby processing node, health check probe information for the active processing node from memory of the standby processing node;
• determining whether the retrieved health check probe information is stale;
• and
• when said health check probe information is determined not to be stale, determining whether or not the active processing node has failed based on the retrieved health check probe information;
• and
• when said health check probe information is determined to be stale, sending, by the standby processing node, a health check probe message from a first health check probing Internet Protocol (IP) communications interface of the standby processing node to a second health check probing IP communications interface of the active processing node.