US 10,122,602 B1Granted

Distributed system infrastructure testing

AssigneeAmazon Technologies, Inc.

InventorsAnuj Prateek, Weston Connor Panther, Christopher Lee Mullins

Life of this patent

Estimating the term…

Loading patent…

Abstract

Techniques are described for performing infrastructure testing of a distributed system. Such testing may be performed by an infrastructure testing service that includes, for example, a manager component and multiple agent components each executing on one of multiple computing devices that are implementing the distributed system. The manager utilizes failure information to schedule failures to occur on target host devices. The manager determines if the distributed system is in a healthy state, and if so, provides failure information to the agent on a target host device. The agent then executes one or more commands on the target host device to cause the failure to occur, and monitors the distributed system and the target host device as they recover from the failure. The infrastructure testing service utilizes this monitored information to initiate other actions based on the recovery.

Claims21

Dependent claims

A computer-implemented method comprising:

receiving, by a computing system of an online program execution service that provides computing resources for use in executing programs of users of the online program execution service and that provides infrastructure test functionality for the users, instructions from one of the users to configure testing of a failure for a distributed system of the user that is implemented using a plurality of computing devices of the provided computing resources and that provides a service over a computer network to end users;
determining, by the computing system and prior to the testing of the failure on a target host device of the plurality, that the distributed system satisfies a specified operational characteristic indicating that the distributed system is available for the testing, including determining that computing devices of the plurality other than the target host device are capable of providing the service for the distributed system during the testing of the target host device;
causing, by the computing system and based on the determining that the distributed system satisfies the operational characteristic, the failure to occur on the target host device during operation of the distributed system to test recovery of the distributed system from the failure, including initiating execution, on the target host device, of a command that implements the failure;
monitoring, by the computing system, the recovery of the distributed system from the failure, and determining a problem in the recovery;
and
initiating, by the computing system and based on the monitoring, an action for correcting the determined problem.

3 more elements

A system comprising:

a processor of a computing system;
and
a memory storing instructions that, when executed by the processor, cause the processor to provide testing of an executing distributed system implemented with computing devices by:
determining, before initiating the testing of the executing distributed system on a target host device that is one of the computing devices, that the executing distributed system satisfies a specified operational characteristic, including determining that at least some of the computing devices other than the target host device are capable of continuing to implement the executing distributed system without the target host device;
causing, based at least in part on the determining, a failure to occur on the target host device to test recovery of the executing distributed system from the failure, including initiating execution of a command on the target host device that alters performance of the target host device;
and
monitoring the recovery of the executing distributed system from the failure, and initiating further actions for the testing based on the monitoring.

4 more elements

A non-transitory computer-readable medium having stored contents that cause a computing system to at least:

determine, by agent software of an infrastructure testing service that is executing on the computing system, one or more commands to execute on the computing system to cause a failure on the computing system to test recoverability of a distributed system, wherein the distributed system is implemented on computing devices that include the computing system;
execute, by the computing system and based on the distributed system satisfying an operational characteristic, the one or more commands to cause the failure on the computing system;
collect, by the computing system and while the failure on the computing system is being caused by execution of the one or more commands, recovery information that describes operations of the distributed system during the failure to attempt to recover from the failure;
halt, by the computing system and while the failure on the computing system continues to be caused by the execution of the one or more commands, and in response to receiving instructions from a user of the distributed system, the execution of the one or more commands and the collection of the recovery information;
and
provide, by the computing system, the collected recovery information to initiate further actions related to the testing that are based on the collected recovery information.

3 more elements

Dependencies are inferred from the claim text.