Coordinated upgrade of a cluster storage system

A distributed storage system comprising:

• one or more storage nodes running node storage system software, wherein at least one of the one or more storage nodes is detected to be running an outdated version of the node storage system software;
• one or more storage devices included within the one or more storage nodes;
• one or more hosts, comprising one or more processors, writing data to and reading data from the one or more storage devices included within the one or more storage nodes, and running host storage system software, wherein at least one of the one or more hosts is detected to be running an outdated version of the host storage system software;
• a host upgrade manager (HUM) installed within at least one of the one or more hosts;
• a primary upgrade manager (PUM) installed within the at least one of the one or more hosts or within at least one of the one or more storage nodes;
• a secondary upgrade manager (SUM) installed within the at least one of the one or more storage nodes;
• wherein the PUM is provided to upgrade the host storage system software and the node storage system software by transferring, via the respective HUM, an upgraded version of the host storage system software to the at least one of the one or more hosts detected to be running the outdated version of the host storage system software and by transferring, via the respective SUM, an upgraded version of the node storage system software to the at least one of the one or more storage nodes detected to be running the outdated version of the node storage system software;
• wherein the host storage system software and the node storage system software control read and write interactions between the one or more hosts and the one or more storage nodes;
• and
• wherein the host storage system software and the node storage system software comprise different, cooperating portions of overall storage management software.

A method for controlling operations of a distributed storage system that includes one or more hosts and one or more storage nodes operating according to host storage system software and node storage system software, respectively, in which the one or more hosts write data to and read data from one or more storage devices included within the one or more storage nodes, the method comprising:

• installing a host upgrade manager (HUM) within at least one of the one or more hosts;
• installing a primary upgrade manager (PUM) within the at least one of the one or more hosts or within at least one of the one or more storage nodes;
• installing a secondary upgrade manager (SUM) within the at least one of the one or more storage nodes;
• running host storage system software and node storage system software, wherein the host storage system software and the node storage system software comprise host storage management code and node storage management code installed within the at least one of the one or more hosts and the at least one of the one or more storage nodes, respectively, wherein the host storage system software and the node storage system software control read and write interactions between the one or more hosts and the one or more storage nodes, and wherein the host storage system software and the node storage system software comprise different, cooperating portions of overall storage management software;
• installing an initial version of the host storage management code within the at least one of the one or more hosts from a source external to both the one or more hosts and the one or more storage nodes;
• determining within any host that the installed initial version of the host storage management code is not current and signaling to the PUM a need for upgrading, pausing read and write operations between the host and any storage node to which the host is connected, offloading selected host data to one of the one or more storage nodes, receiving and installing an upgraded version of the host storage management code from the PUM, reloading the selected offloaded host data, running the upgraded version of the host storage management code, and restarting the read and write operations between the host and any storage node to which the host is connected;
• determining which of the one or more storage nodes are running an outdated version of the node storage management code and are to be upgraded;
• determining whether all SUMs installed within the one or more storage nodes to be upgraded have responded with respective readiness signals within a maximum response time and,
• if so, upgrading the node storage management code in the one or more storage nodes that are ready to be upgraded by transferring, via the respective SUMs, an upgraded version of the node storage management code from the PUM to the respective one or more storage nodes that are ready to be upgraded, and
• if not, performing one of the following preliminary actions:
• deferring upgrading of the one or more storage nodes that are ready to be upgraded,
• stalling the one or more storage nodes that are ready to be upgraded until all of the one or more storage nodes that are ready to be upgraded have signaled readiness, and
• removing from service the one or more storage nodes that are not yet ready for upgrading, determining the respective readiness signals from the one or more storage nodes removed from service and, if a version of the node storage management code in the one or more storage nodes removed from service is not a most recent version, signaling to the one or more storage nodes removed from service to install and operate using the upgraded version of the node storage management code.