Dynamic application instance discovery and state management within a distributed system

A method, comprising:

• storing, by a plurality of discovery and failure detection daemon (DFDD) instances implemented on a plurality of computing devices, state information for a plurality of application instances configured to perform one or more functions in a distributed system;
• updating, by one or more of the plurality of DFDD instances, and based at least in part on information received from a respective application instance, the state information according to a state machine defining transitions between a plurality of states including a state indicating the respective application instance is newly online, a state indicating the respective application instance is operating normally, a state indicating the respective application instance has lost communication with a respective DFDD instance, a state indicating the respective application instance has failed, and a state indicating the respective application instance is subject to a network split;
• selecting, by at least one of the plurality of DFDD instances, a peer DFDD instance of the plurality of DFDD instances;
• and
• communicating, by the at least one of the plurality of DFDD instances, state information for one or more of the plurality of application instances to the peer DFDD instance according to a peer-to-peer synchronization protocol.

A non-transitory computer-accessible storage medium storing instructions that when executed by a computer implement a discovery and failure detection daemon (DFDD) configured to:

• store state information for at least one of a plurality of application instances configured to perform one or more functions in a distributed system;
• update, based at least in part on information received from a respective application instance, the state information according to a state machine defining transitions between a plurality of states including a state indicating the respective application instance is newly online, a state indicating the respective application instance is operating normally, a state indicating the respective application instance has lost communication with a respective DFDD instance, a state indicating the respective application instance has failed, and a state indicating the respective application instance is subject to a network split;
• and
• iteratively perform:
• select one or more peer DFDD instances from among a plurality of DFDD instances;
• and
• execute a peer-to-peer, gossip-based protocol with the one or more peer DFDD instances of the plurality of DFDD instances, wherein to execute the protocol, the one or more peer DFDD instances are configured to exchange state information for at least one of the plurality of application instances in the distributed system.