Systematic testing of failover and recovery for distributed system components

A system for testing failover and recovery comprising:

• one or more state machines, each state machine comprising one or more states, each state machine having a defined role in a distributed system, and each state machine storing a current state of the one or more states in a central storage;
• the central storage to store the current state of the one or more states for each of the one or more state machines;
• and
• a test machine to:
• instantiate the one or more state machines;
• send a termination message to a state machine of the one or more state machines, the termination message causing the state machine to halt the state and the event of the state machine for the one of the system states;
• receive a termination acknowledgment message from the state machine in response to the termination message;
• instantiate a recovery state machine for the halted state machine;
• load the recovery state machine with a same state as the state of the halted state machine and assigning the recovery state machine with a same role as the defined role of the halted state machine and
• verify results of the systematic testing against an expected outcome, the results being generated by running the distributed system with the instantiated recovery state machine for each iteration of the systematic testing;
• and
• a systematic testing framework to perform systematic testing of the distributed system comprising a plurality of iterations, each iteration comprising probabilistically determining a processing order of events for the plurality of state machines in the distributed system to effectuate a system state for the one or more state machines, a system state comprising a combination of states for the one or more state machines, the processing order of events including the termination message.

A computer storage device comprising computer-executable instructions embodied thereon that, when executed by a computing device, test failover and recovery, the instructions causing the computing device to:

• perform systematic testing of a distributed system, the systematic testing probabilistically effectuating system states comprising a combination of states for a plurality of state machines, the systematic testing comprising a plurality of iterations, each iteration testing one the system states, each iteration comprising:
• sending a termination message to a state machine of the plurality of state machines, the state machine having a defined role in the distributed system, the termination message causing the state machine to halt at the state of the state machine in the one of the system states;
• receiving a termination acknowledgment message from the state machine in response to the termination message;
• instantiating a recovery state machine for the halted state machine;
• and loading the recovery state machine with a same state as the state of the halted state machine and assigning the recovery state machine with a same role as the defined role of the halted state machine;
• and
• verify results of the systematic testing against an expected outcome, the results being generated by running the distributed system with the instantiated recovery state machine for each iteration of the systematic testing.