Distributed dynamic system configuration

Apparatus including

• a distributed system including a plurality of nodes, at least one said node including a user interface disposed to receive commands indicating one or more configuration changes associated with said distributed system, each said configuration change including modification of at least one of:
• configuration information associated with said distributed system, configuration information associated with at least one said node;
• said distributed system including a distributed data structure, said data structure indicating one or more tasks to make said configuration changes, each said task including a consistent change to said configuration information, each said task indicating one or more tasklets to perform said task;
• each said tasklet performable at a single said node, each said task indicating one or more target nodes at which to perform a copy of each said tasklet;
• said distributed system including a configuration element, said configuration element disposed to perform copies of each said tasklet asynchronously at substantially all said target nodes, and disposed to identify each said task as complete in response to said performing of copies.

A non-transitory physical medium maintaining information interpretable by a computing device as instructions, said instructions directing said computing device to

• receive commands indicating one or more configuration changes associated with a distributed system including a plurality of nodes, each said configuration change including modification of at least one of:
• configuration information associated with said distributed system, configuration information associated with at least one said node;
• determine one or more tasks to make said configuration changes, each said task including a consistent change to said configuration information;
• determine for each said task one or more tasklets to perform said task, each said tasklet performable at a single said node, and one or more target nodes at which to perform a copy of each said tasklet;
• perform copies of each said tasklet asynchronously at substantially all said target nodes;
• identify each said task as complete in response to a result of said instructions to performing.

A non-transitory physical medium including non-transitory information interpretable by a computing device as instructions, said instructions directing said computing device to

• maintain a distributed system including a plurality of nodes, at least one node including a user interface disposed to receive commands indicating one or more configuration changes associated with said distributed system, each said configuration change including modification of at least one of:
• configuration information associated with said distributed system, configuration information associated with at least one said node;
• maintain in said distributed system a distributed data structure, said data structure indicating one or more tasks to make said configuration changes, each said task including a consistent change to said configuration information, each said task indicating one or more tasklets to perform said task, each said tasklet performable at a single said node, each said task indicating one or more target nodes at which to perform a copy of each said tasklet;
• maintain in said distributed system a configuration element, said configuration element disposed to perform copies of each said tasklet asynchronously at substantially all said target nodes, and disposed to identify each said task as complete in response to said performing of copies.