System and method of providing system jobs within a compute environment

A non-transitory computer-readable apparatus storage medium storing instructions for managing at least a portion of a multi-node compute environment, the instructions which when executed by a processor of a computerized device, perform operations comprising:

• of creating at least one software object, the at least one software object configured to implement a first workload management object event policy;
• associating the at least one software object with at least one queueable workload object;
• monitoring the at least one queueable workload object during, execution thereof, and
• performing, via the at least one software object, of one or more actions configured to provision at least one of (i) the multi-node compute environment, or (ii) a resource external to the multi-node compute environment, the performance of the one or more actions based at least on:
• one or more requirements specified in a workload submission to the multi-node compute environment;
• and
• one or more parameters associated with the at least one queueable workload object meeting a prescribed criterion as detected by the monitoring,
• wherein:
• the creation of the at least one software object comprises creation of a job which is queueable within a queue associated with execution of workload by the multi-node compute environment;
• and
• the association of the at least one software object with at least one queueable workload object comprises association with at least one queueable workload object utilizing a reservation for compute resources within the multi-node compute environment associated therewith.

A method of operating a multi-node compute cluster having a plurality of compute nodes, a workload scheduler process, and a software process operating on resources external to the scheduler process and configured to manage one or more aspects of provisioning of the plurality of compute nodes in support of a user-specified compute workload submission to the multi-node compute cluster, the method comprising:

• creating the software process, the software process configured to implement a workload management object event policy associated with the user-specified compute workload submission;
• monitoring at least one condition specified by the workload management object event policy, the at least one condition associated with at least one of the one or more of the plurality of nodes;
• based at least on the monitoring, performing one or more steps associated with the software process, the one or more steps configured to effect provisioning of the at least one of the one or more of the plurality of nodes to support at least one of a resource requirement or quality requirement specified by the user-specified compute workload submission;
• monitoring for successful completion of the one or more steps;
• and
• releasing the at least one of the one or more of the plurality of nodes to be used for execution of queueable workload objects associated with the user-specified compute workload submission;
• wherein:
• the monitoring for successful completion comprises monitoring a health or failure check of the at least one of the plurality of nodes:
• and
• the releasing the at least one of the one or more of the plurality of nodes comprises causing the releasing only upon successful completion of the health or failure check of the at least one of the plurality of nodes.

Non-transitory computer readable storage medium, the storage medium comprising computerized logic configured to, when executed on a computerized apparatus of a compute cluster having a compute environment comprising a plurality of compute nodes, perform operations comprising:

• creating a software process, the software process configured to implement a workload management object event policy, the workload management object event policy relating to at least one aspect of processing of at least a portion of a compute workload submission to the compute cluster;
• monitoring at least one condition of the compute cluster associated with one or more requirements specified by the workload management object event policy;
• based at least on the monitoring, performing one or more steps associated with the software process, the one or more steps configured to effect provisioning of at least one of the plurality of compute nodes to support the one or more requirements specified by the compute workload submission, the provisioning utilizing at least one resource external to the compute environment;
• monitoring for successful completion of the one or more steps;
• and
• based at least on the successful completion, releasing the at least one of the plurality of compute nodes, the release enabling the at least one of the plurality, of compute nodes to be used for execution of queueable workload objects associated with the compute workload submission,
• wherein:
• the monitoring of the at least one condition of the compute cluster comprises monitoring a health or failure check of at least part of the compute cluster;
• and
• the performing the one or more steps configured to effect provisioning of the at least one of the one or more of the plurality of nodes comprises causing the performance only upon successful completion of the health or failure check of the compute cluster.