Fault-tolerant and highly available configuration of distributed services

One or more non-transitory computer-readable media storing one or more programs for execution by one or more processors, the one or more programs comprising instructions for:

• obtaining, at a first host, service-host binding information from a replica of a distributed configuration store;
• determining, at the first host, one or more services to watch based on the service-host binding information, a service of the one or more services being a single instance of a software product or software application installed on one or more hosts in a distributed computing environment;
• setting a watch on the one or more services to watch;
• receiving a notification that a target state of a particular service of the one or more services to watch is modified, comprising updating target state information in the replica of the distributed configuration store using a consensus protocol that allows hosts to agree on an ordering for changes in the distributed configuration store;
• wherein the target state of the particular service represents a desired operational state for the particular service;
• determining, at the first host, a current state of the particular service of the one or more services to watch, wherein the current state represents an operational state for the particular service, and wherein the particular service is hosted at the first host;
• determining whether the current state of the particular service is inconsistent with the target state of the particular service of the one or more services to watch;
• in response to determining that the current state of the particular service is inconsistent with the target state of the particular service, the first host taking a corrective action to modify the particular service, which causes the current state of the particular service to be consistent with the target state of the particular service.

A system, comprising:

• one or more processors;
• one or more storage media storing one or more programs for execution by the one or more processors, the one or more programs comprising instructions for:
• obtaining, at a first host of one or more hosts in a distributed computing environment, service-host binding information from a replica of a distributed configuration store, wherein the replica of the distributed configuration store represents a local copy of the service-host binding information;
• determining, at the first host, one or more services to watch based on the service-host binding information, a service of the one or more services being a single instance of a software product or software application installed on at least the first host or one of the one or more hosts in a distributed computing environment;
• setting a watch on the one or more services to watch;
• based on the watch, receiving a notification that a target state of a particular service of the one or more services to watch is modified, comprising updating target state information in the replica of the distributed configuration store using a consensus protocol that allows hosts to agree on an ordering for changes in the distributed configuration store;
• wherein the updated target state of the particular service represents a desired operational state for the particular service;
• determining, at the first host, a current state of the particular service of the one or more services to watch, wherein the current state represents an operational state for the particular service, and wherein the particular service is hosted at the first host;
• determining whether the current state of the particular service is inconsistent with the updated target state of the particular service of the one or more services to watch;
• in response to determining that the current state of the particular service is inconsistent with the updated target state of the particular service, the first host taking a corrective action to modify the particular service, which causes the current state of the particular service to be consistent with the updated target state of the particular service.