Optimizing job preemption using dynamic and static information

A computer program product comprising:

• one or more computer-readable storage media and program instructions collectively stored on the one or more computer-readable storage media, the stored program instructions comprising:
• program instructions to responsive to a pending job being selected to be run, determine that no computing hosts within a computing cluster have available resources capable of running the pending job and non-preemptive measures will not provide required resources for the pending job, wherein determining that no computing hosts within the computing cluster have available resources capable of running the pending job and the non-preemptive measures will not provide the required resources for the pending job comprises:
• program instructions to review current jobs running and available resources within the computing cluster at a point in time by taking a snapshot;
• and program instructions to compare the available resources within the computing cluster to the required resources to run the pending job;
• program instructions to identify candidate jobs for preemption and dynamic information and static information for each candidate job, wherein the static information is a configured priority value assigned to a respective candidate job and pulled from a job submission script, wherein the dynamic information is identified by polling the computing cluster for the dynamic information about each candidate job as that candidate job runs in the computing cluster, wherein the one or more processors maintain, in a job scheduler component:
• (i) a first data structure that stores, for each running job within the computing cluster, a latest checkpoint time detected for that running job, the latest checkpoint time being recorded explicitly using a scheduler application programming interface (API) by which the running job marks successful completion of a checkpoint, and
• (ii) a second data structure that stores, for each running job within the computing cluster, a predicted total time for completion and a start time for that running job, wherein the dynamic information includes:
• an elapsed time from a last checkpoint of a respective candidate job, calculated as a difference between a current time and the latest checkpoint time recorded in the first data structure for the respective candidate job and
• a predicted remaining time for completion of the respective candidate job, calculated as a difference between the predicted total time for completion recorded in the second data structure for the respective candidate job,
• the start time recorded in the second data structure for the respective candidate job, and
• the current time, wherein the predicted total time for completion is predicted implicitly by applying job profiling based on resources needed and resources available to predict a total time for completion of the respective candidate job using extrapolation or one or more machine learning models;
• program instructions to rank the candidate jobs for preemption based on the dynamic information and the static information for each candidate job, wherein ranking the candidate jobs for preemption is done with an objective of minimizing loss of computation time while optimizing a benefit and utilization of resources of the computing cluster, and wherein ranking the candidate jobs for preemption comprises utilizing a unified metric of the dynamic information and the static information for scoring each candidate job, the unified metric being:
• Mj=WCD*CDj+WLD*LDj+WPS*PS(j),
• wherein WCD, WLD, and WPS are weights for aspect functions of candidate job j. wherein CD is a first function that produces a first score such that the first score is higher as the elapsed time from the last checkpoint is lower,
• wherein LD is a second function that produces a second score such that the second score is higher as the predicted remaining time is higher, and
• wherein PS is a third function that produces a third score such that the third score is higher as the configured priority value is lower;
• program instructions to attempt to preempt a top N candidate jobs whose released resources in combination with the available resources of the computing cluster satisfy the required resources of the pending job;
• and
• program instructions to responsive to successfully preempting the top N candidate jobs, initiate the pending job.

A computer system comprising:

• one or more computer processors;
• one or more computer-readable storage media;
• program instructions collectively stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors, the stored program instructions comprising:
• program instructions to responsive to a pending job being selected to be run, determine that no computing hosts within a computing cluster have available resources capable of running the pending job and non-preemptive measures will not provide required resources for the pending job, wherein determining that no computing hosts within the computing cluster have available resources capable of running the pending job and the non-preemptive measures will not provide the required resources for the pending job comprises:
• program instructions to review current jobs running and available resources within the computing cluster at a point in time by taking a snapshot;
• and program instructions to compare the available resources within the computing cluster to the required resources to run the pending job;
• program instructions to identify candidate jobs for preemption and dynamic information and static information for each candidate job, wherein the static information is a configured priority value assigned to a respective candidate job and pulled from a job submission script, wherein the dynamic information is identified by polling the computing cluster for the dynamic information about each candidate job as that candidate job runs in the computing cluster, wherein the one or more processors maintain, in a job scheduler component:
• (i) a first data structure that stores, for each running job within the computing cluster, a latest checkpoint time detected for that running job, the latest checkpoint time being recorded explicitly using a scheduler application programming interface (API) by which the running job marks successful completion of a checkpoint, and
• (ii) a second data structure that stores, for each running job within the computing cluster, a predicted total time for completion and a start time for that running job, wherein the dynamic information includes:
• an elapsed time from a last checkpoint of a respective candidate job, calculated as a difference between a current time and the latest checkpoint time recorded in the first data structure for the respective candidate job and
• a predicted remaining time for completion of the respective candidate job, calculated as a difference between the predicted total time for completion recorded in the second data structure for the respective candidate job,
• the start time recorded in the second data structure for the respective candidate job, and
• the current time, wherein the predicted total time for completion is predicted implicitly by applying job profiling based on resources needed and resources available to predict a total time for completion of the respective candidate job using extrapolation or one or more machine learning models;
• program instructions to rank the candidate jobs for preemption based on the dynamic information and the static information for each candidate job, wherein ranking the candidate jobs for preemption is done with an objective of minimizing loss of computation time while optimizing a benefit and utilization of resources of the computing cluster, and wherein ranking the candidate jobs for preemption comprises utilizing a unified metric of the dynamic information and the static information for scoring each candidate job, the unified metric being:
• Mj=WCD*CDj+WLD*LDj+WPS*PS(j),
• wherein WCD, WLD, and WPS are weights for aspect functions of candidate job j, wherein CD is a first function that produces a first score such that the first score is higher as the elapsed time from the last checkpoint is lower,
• wherein LD is a second function that produces a second score such that the second score is higher as the predicted remaining time is higher, and
• wherein PS is a third function that produces a third score such that the third score is higher as the configured priority value is lower;
• program instructions to attempt to preempt a top N candidate jobs whose released resources in combination with the available resources of the computing cluster satisfy the required resources of the pending job;
• and
• program instructions to responsive to successfully preempting the top N candidate jobs, initiate the pending job.