System and method for upgrading sparkline cluster with zero downtime

A non-transitory computer-readable medium including stored thereon a sequence of instructions which, when executed by a processor causes the processor to execute a process, the process comprising:

• obtaining a request to update a first dataflow cluster, the first dataflow cluster directing query data from a gateway node to a first series of executor nodes to execute queries via a first driver node;
• generating a second dataflow cluster, generating the second dataflow cluster comprising:
• establishing a second series of executor nodes from a new node pool, the second series of executor nodes comprise cache data mapping file system data for segment caching;
• and
• establishing a second driver node configured to forward queries from the gateway node to the second series of executor nodes;
• causing a first update to a configuration of a host configuration node to register the second dataflow cluster as an active endpoint for subsequent queries and identify the first dataflow cluster as an inactive endpoint;
• and
• responsive to determining that no active queries exist, causing a second update to the configuration of the host configuration node to remove the first dataflow cluster from the configuration, causing direction of subsequent query data from the gateway node to the second dataflow cluster.

A system comprising:

• a processor;
• and
• a memory including a sequence of instructions that, when executed by the processor causes the processor to execute a process, the process comprising:
• obtaining a request to update a first dataflow cluster configured to execute queries provided by a gateway node;
• generating a second dataflow cluster as a replacement to the first dataflow cluster, the second dataflow cluster comprising a second series of executor nodes, the second series of executor nodes configured to execute subsequent queries provided from the gateway node;
• causing a first update to a configuration to register the second dataflow cluster as an active endpoint for subsequent queries and identify the first dataflow cluster as an inactive endpoint;
• and
• responsive to determining that no active queries exist, causing a second update to the configuration to remove the first dataflow cluster from the configuration, causing direction of subsequent queries from the gateway node to the second dataflow cluster.