Job execution with managed compute environments

A computer-implemented method, comprising:

• providing an interface to receive compute environment service requests;
• receiving, from a client via the interface, a compute environment service request indicating a type of compute environment, wherein the compute environment management system supports managed and unmanaged compute environments;
• receiving, from the client, a specification to allow compute instances of a spot resources type to be included in the managed compute environment during batch processing;
• determining whether the type of compute environment is for an unmanaged compute environment or a managed compute environment, wherein the managed compute environment automatically adds or removes one or more compute instances within the managed compute environment as workload for the managed compute environment changes during the batch processing;
• based on a determination that the type of compute environment is for a managed compute environment, establishing the managed compute environment according to specified constraints for compute instances, wherein the specified constraints are received from the client and include:
• the specification to allow compute instances of a spot resources type to be included in the managed compute environment and indicate a desired total number of compute instances to be allocated in the managed compute environment, a specified computational capacity of the one or more compute instances, and a specified software stack for the one or more compute instances;
• subsequently, receiving a batch of jobs to be executed in the managed compute environment, wherein a given job of the batch of jobs comprises an indication that:
• the given job is allowed to be executed on a compute instance of a spot resource type, and
• the given job is to be terminated after a maximum time has elapsed;
• selecting, based on the indication that the given job is allowed to be executed on the compute instance of the spot resource type, one or more compute instances for the managed compute environment, including a given compute instance of the spot resource type, to be used to execute the batch of jobs, wherein the selection is according to requirements for the given job and the specified constraints;
• provisioning the selected one or more compute instances in the managed compute environment;
• initiating execution of the batch of jobs on or across the one or more compute instances in parallel in the managed compute environment for the batch processing;
• and
• automatically scaling, during the execution of the batch of jobs, based on number of pending jobs of the batch of jobs and the maximum time, a number of compute instances allocated to the managed compute environment.

One or more non-transitory computer-readable storage media storing instructions that, when executed on or across one or more processors, cause the one or more processors to:

• provide an interface to receive compute environment service requests;
• receive, from a client via the interface, a compute environment service request indicating a type of compute environment, wherein the compute environment management system supports managed and unmanaged compute environments;
• receive, from the client, a specification to allow compute instances of a spot resources type to be included in the managed compute environment during batch processing;
• determine whether the type of compute environment is for an unmanaged compute environment or a managed compute environment, wherein the managed compute environment automatically adds and removes one or more compute instances within the managed compute environment as workload for the managed compute environment changes during the batch processing;
• based on a determination that the type of compute environment is for a managed compute environment, establish the managed compute environment according to specified constraints for compute instances, wherein the specified constraints are received from the client and include:
• the specification to allow compute instances of a spot resources type to be included in the managed compute environment and indicate a desired total number of compute instances to be allocated in the managed compute environment, a specified computational capacity of the one or more compute instances, and a specified software stack for the one or more compute instances;
• subsequently, receive a batch of jobs to be executed in the managed compute environment, wherein a given job of the batch of jobs comprises an indication that:
• the given job is allowed to be executed on a compute instance of a spot resource type, and
• the given job is to be terminated after a maximum time has elapsed;
• select, based on the indication that the given job is allowed to be executed on the compute instance of the spot resource type, one or more compute instances for the managed compute environment, including a given compute instance of the spot resource type, to be used to execute the batch of jobs, wherein the selection is according to requirements for the given job and the specified constraints;
• provision the selected one or more compute instances in the managed compute environment;
• initiate execution of the batch of jobs on or across the one or more compute instances in parallel in the managed compute environment for the batch processing;
• and
• automatically scale, during the execution of the batch of jobs, based on number of pending jobs of the batch of jobs and the maximum time, a number of compute instances allocated to the managed compute environment.