Systems and methods for testing resilience of a distributed network

A resilience testing system for testing the resilience of a distributed network, the distributed network comprising one or more scalable applications, each scalable application comprising a plurality of compute groups, each compute group comprising one or more active compute instances, the resilience testing system comprising:

• a test database for storing test parameters corresponding to each of the one or more scalable applications, the test parameters indicating at least a schedule for performing a resilience test on the scalable application, unique identifiers of the plurality of compute groups registered for resilience testing, and a probability of terminating a compute instance, wherein each respective compute group in the plurality of compute groups is assigned a respective probability of terminating the compute instance;
• a controller for scheduling a task based on the schedule for performing the resilience test on the one or more scalable applications, the task including a task descriptor comprising at least the unique identifiers of the plurality of compute groups registered for resilience testing, and the respective probability of terminating a compute instance;
• a processing node for retrieving the task descriptor for the resilience test and programmed for:
• determining, prior to retrieving compute group data related to the resilience test from the distributed network, whether to terminate a compute instance or not based on the respective probability of terminating the compute instance;
• in response to determining to terminate the compute instance:
• selecting a compute group from the plurality of compute groups associated with the scalable application, the selected compute group associated with the compute instance;
• requesting, from the distributed network, a list of active compute instances for the selected compute group;
• randomly selecting an active compute instance from the list of active compute instances for terminating;
• causing the selected active compute instance to terminate.

A non-transitory computer readable medium comprising instructions which, when executed by a processor, cause the processor to:

• initiate a scheduled resilience test for a scalable application offered by a distributed network, the distributed network offering one or more scalable applications, each scalable application associated with a plurality of compute groups, each compute group comprising one or more active compute instances;
• retrieve a task descriptor comprising test parameters associated with the scheduled resilience test for the scalable application, the test parameters indicating at least a schedule for performing the scheduled resilience test on the scalable application, unique identifiers of the plurality of compute groups registered for resilience testing, and a probability of terminating a compute instance, wherein each respective compute group in the plurality of compute groups is assigned a respective probability of terminating the compute instance;
• and
• determine, prior to retrieving compute group data related to the scheduled resilience test from the distributed network, whether to terminate a compute instance or not based on the respective probability of terminating the compute instance and in response to determining to terminate the compute instance:
• select a compute group from the plurality of compute groups associated with scalable application, the selected compute group associated with the compute instance;
• request, from the distributed network, a list of active compute instances for the selected compute group;
• randomly select an active compute instance from the list of active compute instances for terminating;
• cause the selected active compute instance to terminate.