Automating and monitoring rolling cluster reboots

A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device cause the at least one processing device:

• to initiate a rolling cluster reboot for a cluster comprising a plurality of processing nodes;
• to obtain encoded state information specifying a plurality of states for transitioning a given processing node of the cluster from running as part of the cluster to rebooting and back to running as part of the cluster, the encoded state information further specifying at least one timeout associated with a transition to a given one of the plurality of states;
• to monitor transitioning of the given processing node between the plurality of states specified in the encoded state information;
• to generate an alert responsive to determining that an elapsed time has exceed the at least one timeout specified in the encoded state information without the given processing node having transitioned to the given state;
• to provide the alert to a given client device coupled to the cluster via at least one network;
• and
• to repeat the monitoring, generating and providing while one or more other processing nodes in the cluster transition, in a sequence, between the plurality of states specified in the encoded state information as part of the rolling cluster reboot;
• wherein monitoring the transitioning of the given processing node between the plurality of states specified in the encoded state information is performed by at least one other processing node in the cluster;
• wherein the at least one other processing node in the cluster periodically requests the given processing node to report a current state of the plurality of states;
• and
• wherein generating the alert is responsive to the at least one other processing node determining that an elapsed time for the current state of the plurality of states exceeds the at least one timeout.

An apparatus comprising:

• at least one processing device comprising a processor coupled to a memory;
• the at least one processing device being configured:
• to initiate a rolling cluster reboot for a cluster comprising a plurality of processing nodes;
• to obtain encoded state information specifying a plurality of states for transitioning a given processing node of the cluster from running as part of the cluster to rebooting and back to running as part of the cluster, the encoded state information further specifying at least one timeout associated with a transition to a given one of the plurality of states;
• to monitor transitioning of the given processing node between the plurality of states specified in the encoded state information;
• to generate an alert responsive to determining that an elapsed time has exceed the at least one timeout specified in the encoded state information without the given processing node having transitioned to the given state;
• to provide the alert to a given client device coupled to the cluster via at least one network;
• and
• to repeat the monitoring, generating and providing while one or more other processing nodes in the cluster transition, in a sequence, between the plurality of states specified in the encoded state information as part of the rolling cluster reboot;
• wherein monitoring the transitioning of the given processing node between the plurality of states specified in the encoded state information is performed by at least one other processing node in the cluster;
• wherein the at least one other processing node in the cluster periodically requests the given processing node to report a current state of the plurality of states;
• and
• wherein generating the alert is responsive to the at least one other processing node determining that an elapsed time for the current state of the plurality of states exceeds the at least one timeout.