Maximizing spot savings with live migration and pre-provisioned nodes

A non-transitory computer readable storage medium having instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to perform steps comprising:

• provisioning a plurality of spot instances based on corresponding node templates, wherein each of the plurality of spot instances is a virtual machine provisioned from a cloud provider, and is subject to interruption by the cloud provider based on dynamic capacity condition at the cloud provider;
• instantiating a plurality of production workloads on the plurality of spot instances;
• generating, for each node template, a standby node template that mirrors a corresponding node template;
• provisioning one or more on-demand instances as standby instances based on corresponding standby node templates, wherein each of the one or more on-demand instances is a virtual machine provisioned from the cloud provider, and is not subject to interruption by the cloud provider based on dynamic capacity condition at the cloud provider;
• instantiating a plurality of dummy workloads on the on-demand instances, each dummy workload pre-pulling a container image and reserving compute resources for a corresponding production workload;
• monitoring the plurality of spot instances to detect interruptions;
• in response to detecting an interruption of a spot instance that runs a production workload:
• identifying a dummy workload on a standby node that corresponds to the production workload on the interrupted spot instance,
• applying a temporary lock to the standby node to prevent other workloads from being scheduled on the standby node during migration, and
• migrating the production workload to the standby node.

A system, comprising:

• one or more processors;
• and
• a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by the one or more processors, cause the one or more processors to perform steps comprising:
• provisioning a plurality of spot instances based on corresponding node templates, wherein each of the plurality of spot instances is a virtual machine provisioned from a cloud provider, and is subject to interruption by the cloud provider based on dynamic capacity condition at the cloud provider;
• instantiating a plurality of production workloads on the plurality of spot instances;
• generating, for each node template, a standby node template that mirrors a corresponding node template;
• provisioning one or more on-demand instances as standby instances based on corresponding standby node templates, wherein each of the one or more on-demand instances is a virtual machine provisioned from the cloud provider, and is not subject to interruption by the cloud provider based on dynamic capacity condition at the cloud provider;
• instantiating a plurality of dummy workloads on the on-demand instances, each dummy workload pre-pulling a container image and reserving compute resources for a corresponding production workload;
• monitoring the plurality of spot instances to detect interruptions;
• in response to detecting an interruption of a spot instance that runs a production workload:
• identifying a dummy workload on a standby node that corresponds to the production workload on the interrupted spot instance,
• applying a temporary lock to the standby node to prevent other workloads from being scheduled on the standby node during migration, and
• migrating the production workload to the standby node.