Coordinated upgrades in distributed systems

A cloud computing platform comprising:

• a plurality of host devices, each of said host devices comprising a processor;
• for each of said host devices, at least one virtual machine having a guest operating system;
• for each of said virtual machine, at least one replica of a distributed application, said distributed application comprising a plurality of said replicas;
• a fabric controller that:
• determines a grouping for said replicas, said grouping defining a plurality of update domains, each of said update domains comprising a plurality of said replicas;
• queries the distribution application to determine a sequence of said update domains to update;
• processes each of said update domains according to said sequence by processing a current update domain in an atomic fashion, said current update domain being processed by a method comprising:
• querying said application executing in a current update domain, before transmission of said update for reception by said current update domain, to determine whether said application considers said current update domain ready for updating;
• after receiving a query response indicative of said current update domain being ready for updating, performing said update on said update domain;
• querying said application executing in said current update domain to determine if said update results in said application operating properly within said current update domain;
• when said update results in said application operating properly, proceeding to a next update domain;
• and
• when said update results in said application operating improperly, suspending said update.

A method performed on a plurality of computer processors, said method comprising:

• executing a distributed application on said plurality of computer processors, said distributed application being executed by a plurality of replicas of application code;
• said distributed application being executed on virtual machines having guest operating systems, said virtual machines being operated on host devices;
• identifying an update to perform on said distributed application;
• transmitting information about said update to said distributed application to be considered by said distributed application in connection with responding to queries regarding said update;
• querying said distributed application to determine a grouping of said plurality of replicas into update domains for said update, said update domains being groups of said replicas;
• querying said distributed application, by a fabric controller, to determine a sequence of said update domains to update;
• for each of said update domains, processing, by the fabric controller according to the sequence, a current update domain to perform a first step of said update, said processing being performed as a first atomic operation and comprising:
• querying said application executing in a current update domain, before transmission of said update for reception by said current update domain, to determine whether said application considers said current update domain ready for updating;
• and
• if said application is not ready for updating, performing a repair on said application within said current update domain;
• for each of said update domains, processing a current update domain to perform a second step of said update, said processing being performed as a second atomic operation and comprising:
• after receiving a query response indicative of said current update domain being ready for updating, performing said update on said update domain;
• after performing said update, querying said application executing in a current update domain to determine whether said current update domain is operating properly;
• if said application is operating properly, completing said second step on said current update domain.