Job scheduling based on historical job data

A data analyzer, comprising:

• a processor;
• and
• a memory storing instructions which, when executed by the processor, cause the data analyzer to:
• access a database storing historical job data of job submissions received from a plurality of users, the database comprising resource request data indicative of computing resources requested by each of the users for executing jobs, and resource usage data indicative of actual computing resources used by each of the users to execute the jobs;
• calculate a score for each of the plurality of users by comparing the resource request data and the resource usage data, the score indicating a degree to which a user overestimates computing resources required to execute the jobs;
• receive a new job submission from a user of the plurality users, the new job submission specifying a new job to be executed and computing resources requested by the user for executing the new job;
• calculate a utilization factor for the new job being a ratio of the computing resources requested by the user for executing the new job to computing resources available during a backfill window;
• schedule the new job for execution using the score of the user and the utilization factor of the new job;
• and
• execute the new job on a subset of the available computing resources during the backfill window, wherein the backfill window is a period of time in the future that the new job will execute and wherein the subset of the available computing resources comprises fewer computing resources than the requested computing resources.

A data analysis apparatus, comprising:

• a database storing historical job data of job submissions received from a plurality of users, comprising resource request data indicative of computing resources requested by each of the users for executing jobs, and resource usage data indicative of actual computing resources used by each of the users to execute the jobs;
• and
• a data analyzer, comprising;
• a processor;
• and
• a memory storing instructions which, when executed by the processor, cause the data analyzer to:
• access the database;
• calculate a score for each of the plurality of users by comparing the resource request data and the resource usage data, the score indicating a degree to which a user overestimates computing resources required to execute the job;
• receive a new job submission from a user of the plurality users, the new job submission specifying a new job to be executed and computing resources requested by the user for executing the new job;
• calculate a utilization factor for the new job being a ratio of the computing resources requested by the user for executing the new job to computing resources available during a backfill window;
• and
• schedule the new job for execution using the score of the user and the utilization factor of the new job;
• and
• execute the new job on a subset of the available computing resources during the backfill window, wherein the backfill window is a period of time in the future that the new job will execute and wherein the subset of the available computing resources comprises fewer computing resources than the requested computing resources.

A computer system, comprising:

• a database storing historical job data of job submissions received from a plurality of users, comprising resource request data indicative of computing resources requested by each of the users for executing jobs, and resource usage data indicative of actual computing resources used by each of the users to execute the job;
• a data analyzer, comprising:
• a processor;
• and
• a memory storing instructions which, when executed by the processor, cause the data analyzer to:
• access the database;
• calculate a score for each of the plurality of users by comparing the resource request data and the resource usage data, the score indicating a degree to which a user overestimates computing resources required to execute the job;
• receive a new job submission from a user of the plurality users, the new job submission specifying a new job to be executed and computing resources requested by the user for executing the new job;
• and
• calculate a utilization factor for the new job being a ratio of the computing resources requested by the user for executing the new job to computing resources available during a backfill window;
• a scheduler configured to schedule jobs including the new job using the score of the user and the utilization factor of the new job;
• and
• computing resources configured to execute scheduled jobs including executing the new job on a subset of the available computing resources during the backfill window, wherein the backfill window is a period of time in the future that the new job will execute and wherein the subset of the available computing resources comprises fewer computing resources than the computing resources requested by the user.

A non-transitory computer readable medium storing computer-readable code which, when executed by a computer, causes the computer to perform a method of job scheduling in a computing environment, the method comprising:

• collecting historical job data of job submissions received from a plurality of users, the historical job data comprising resource request data indicative of computing resources requested by each of the users for executing jobs, and resource usage data indicative of actual computing resources used by each of the users to execute the jobs;
• calculating a score for each of the plurality of users by comparing the resource request data and the resource usage data, the score indicating a degree to which a user overestimates computing resources required to execute job;
• receiving a new job submission from a user, the new job submission specifying a new job to be executed and computing resources requested by the user for executing the new job;
• calculating a utilization factor for the new job being a ratio of the computing resources requested by the user for executing the new job to the computing resources available during a backfill window;
• scheduling the new job for execution using the score of the user and the utilization factor of the new job;
• and
• executing the new job on a subset of the available computing resources during the backfill window, wherein the backfill window is a period of time in the future that the new job will execute and wherein the subset of the available computing resources comprises fewer computing resources than the computing resources requested by the user.