Scheduling data access jobs based on job priority and predicted execution time using historical execution data

A system, comprising:

• at least one memory storing computer-executable instructions;
• and
• at least one processor in communication with the at least one memory, the at least one processor configured to access the at least one memory and execute the computer-executable instructions to:
• receive a job request describing a primary data access job to be executed to access data stored in at least one of a plurality of data storage systems, the job request including a target completion duration for requested completion of the primary data access job;
• identify one or more preliminary data access jobs such that execution of the primary data access job depends on a result of the one or more preliminary data access jobs;
• based on historical data describing at least one previous execution of the primary data access job and the one or more preliminary data access jobs, predicting execution durations of the primary data access job and the one or more preliminary data access jobs;
• determine one or more subsets of the one or more preliminary data access jobs, wherein individual ones of the one or more subsets include serially dependent preliminary data access jobs and are characterized by a subset execution duration that is based on a sum of the predicted execution durations of the serially dependent preliminary data access jobs of a subset of the one or more subsets, and a total execution duration of the individual ones of the subsets characterized by a sum of the subset execution duration of the serially dependent preliminary data access jobs of the subset of the one or more subsets and predicted execution duration of the primary data access job;
• determining a time-sensitive subset of the one or more subsets, the time-sensitive subset having a smallest time difference between the target completion duration for completion of the requested primary data access job and the total execution duration of the individual ones of the subsets;
• and
• schedule a start time for at least one of the one or more preliminary data access jobs to begin execution and upon completion to provide the result of the one or more preliminary data access jobs to the primary data access job prior to a start time of the primary data access job so the primary data access job completes by the target completion duration and the scheduling including prioritizing at least one of the preliminary data access jobs included in the time-sensitive subset to execute prior to at least one other preliminary data access jobs included in the time-sensitive subset to execute prior to at least one other preliminary data access job not included in the time-sensitive subset and adjust the scheduling of the start time based on real time monitoring of status of the one or more preliminary data access jobs.

One or more non-transitory computer-readable media storing instructions which, when executed by at least one processor, instruct the at least one processor to perform actions comprising:

• in response to receiving a job request, identifying one or more preliminary data access jobs to be executed prior to execution of a primary data access job, the job request describing the primary data access job to be executed to access data stored in one or more data storage systems, the primary data access job with a target completion duration;
• based on historical data describing at least one previous execution of the primary data access job and the one or more preliminary data access jobs, predicting execution durations of the primary data access job and the one or more preliminary data access jobs;
• determining one or more subsets of the one or more preliminary data access jobs, wherein individual ones of the one or more subsets include serially dependent preliminary data access jobs and are characterized by a subset execution duration that is based on a sum of the predicted execution durations of the serially dependent preliminary data access jobs of a subset of the one or more subsets, and a total execution duration of the individual ones of the subsets characterized by a sum of the subset execution duration of the serially dependent preliminary data access jobs of the subset of the one or more subsets and the predicted execution duration of the primary data access job;
• determining a time-sensitive subset of the one or more subsets, the time-sensitive subset having a smallest time difference between the target completion duration for requested completion of the primary data access job and the total execution duration of the individual ones of the subsets;
• and
• scheduling start times for the one or more preliminary data access jobs to begin execution and upon completion to provide the result of the one or more preliminary data access jobs to the primary data access job prior to the start time of the primary data access job so the primary data access job completes by the target completion duration, and the scheduling including prioritizing at least one of the preliminary data access jobs included in the time-sensitive subset to execute prior to at least one other preliminary data access job not included in the time-sensitive subset and adjusting the scheduling of the start times based on real time monitoring of status of the one or more preliminary data access jobs.