Dynamic time slicing for data-processing workflow

A concurrent scheduling system for dynamically scheduling a data-processing workflow, comprising:

• a plurality of worker computing devices, each worker computing device configured to execute one or more concurrent worker processes, wherein the plurality of worker computing devices collectively execute a plurality of concurrent worker processes;
• and
• a job server computing device configured to:
• recognize a minimum execution slice size representing a minimum duration of date-dependent records for processing in a job, and a maximum execution slice size representing a maximum duration of date-dependent records for processing in a job;
• and
• maintain a collection of jobs as a concurrent data structure configured for concurrent reading and writing by a plurality of worker processes;
• wherein each worker process is configured, concurrently with the other worker processes, to:
• select a current job from the collection of jobs;
• recognize a predicted execution slice size for the current job based on a duration defined by a start date and an end date for date-dependent records associated with the current job;
• responsive to one or both of 1) the predicted execution slice size for the current job exceeding the maximum execution slice size, and 2) the end date for the current job being in the future of a current date:
• update the collection of jobs to indicate that the current job is locked;
• split the current job into a working slice and a remainder slice;
• update the collection of jobs to include the remainder slice;
• execute the working slice;
• and
• update the collection of jobs to indicate that the working slice is completed;
• and
• responsive to the predicted execution slice size for the current job exceeding the minimum execution slice size and not exceeding the maximum execution slice size:
• update the collection of jobs to indicate that the current job is locked;
• execute the current job;
• and
• update the collection of jobs to indicate that the current job is completed.

A method for dynamically scheduling a data-processing workload, the method comprising:

• recognizing a minimum execution slice size representing a minimum duration of date-dependent records for processing in a job, and a maximum execution slice size representing a maximum duration of date-dependent records for processing in a job;
• and
• maintaining a collection of jobs as a concurrent data structure configured for concurrent reading and writing by a plurality of worker processes;
• wherein each worker process is configured, concurrently with the other worker processes, to:
• select a current job from the collection of jobs;
• recognize a predicted execution slice size for the current job based on a duration defined by a start date and an end date for date-dependent records associated with the current job;
• responsive to one or both of 1) the predicted execution slice size for the current job exceeding the maximum execution slice size, and 2) the end date for the current job being in the future of a current date:
• update the collection of jobs to indicate that the current job is locked;
• split the current job into a working slice and a remainder slice;
• update the collection of jobs to include the remainder slice;
• execute the working slice;
• and
• update the collection of jobs to indicate that the working slice is completed;
• and
• responsive to the predicted execution slice size for the current job exceeding the minimum execution slice size and not exceeding the maximum execution slice size:
• update the collection of jobs to indicate that the current job is locked;
• execute the current job;
• and
• update the collection of jobs to indicate that the current job is completed.