Back to results
US 10,372,492 B2Granted
Job-processing systems and methods with inferred dependencies between jobs
AssigneeDropbox, Inc.
InventorsShaunak Kishore, Karl Dray
Life of this patent
Estimating the term…
Abstract
An analytics system that executes processing jobs infers dependencies between jobs to be executed based on identification of dependencies between a “sink” job and a source data object on which the sink job depends. Given a job definition for the sink job that identifies a source data object, the system can identify a “source” job that produces the source data object and can infer a dependency of the sink job on the source job. The system can schedule executions of the source and sink jobs such that the source job completes (or completes generation of the source data object) before the sink job is launched.
Claims15
A computer-implemented method comprising:
- receiving, at a computer system, via graphical user interface controls of a job creation graphical user interface, a plurality of job definitions for a plurality of jobs, each job definition of the plurality of job definitions specifying at least a source data object to provide input data to be consumed by the job, and an output data object to store output data produced by the job;
- wherein the plurality of job definitions includes a first job definition and a second job definition;
- wherein the plurality of jobs includes a first job and a second job;
- wherein the first job definition is for the first job;
- wherein the second job definition is for the second job;
- automatically inferring, by the computer system, a first dependency of the first job on the second job;
- wherein the automatically inferring the first dependency is based at least in part on automatically determining that the source data object specified in the first job definition is the output data object specified in the second job definition;
- based at least in part on the automatically inferring the first dependency, dispatching, by the computer system, a first plurality of job instances for execution, wherein the dispatching the first plurality of job instances is controlled such that a first instance of the first job, of the first plurality of job instances, is blocked from being dispatched until after a first instance of the second job, of the first plurality of job instances, completes;
- after the dispatching the first plurality of job instances for execution, receiving, at the computer system, via graphical user interface controls of the job creation graphical user interface, a third job definition;
- wherein the third job definition is for a third job;
- wherein the third job definition specifies at least a source data object to provide input data to be consumed by the third job, and an output data object to store output data produced by the third job;
- based at least on the third job definition, automatically inferring, by the computer system, a second dependency;
- wherein the second dependency is the first job on the third job;
- wherein the second dependency is automatically inferred based at least in part on automatically determining that the source data object specified in the first job definition is the output data object specified in the third job definition;
- and
- based at least in part on the automatically inferring the first dependency and the automatically inferring the second dependency, dispatching, by the computer system, a second plurality of job instances for execution, wherein the dispatching the second plurality of job instances is controlled such that a second instance of the first job, of the second plurality of job instances, is blocked from being dispatched until after both:
- (a) a second instance of the second job, of the second plurality of job instances, completes, and (b) a first instance of the third job, of the second plurality of job instances, completes.
14 more elements
One or more non-transitory computer-readable media storing one or more programs for execution by one or more processing units, the one or more programs comprising instructions configured for:
- receiving, at a computer system, via graphical user interface controls of a job creation graphical user interface, a plurality of job definitions for a plurality of jobs, each job definition of the plurality of job definitions specifying at least a source data object to provide input data to be consumed by the job, and an output data object to store output data produced by the job;
- wherein the plurality of job definitions includes a first job definition and a second job definition;
- wherein the plurality of jobs includes a first job and a second job;
- wherein the first job definition is for the first job;
- wherein the second job definition is for the second job;
- automatically inferring, by the computer system, a first dependency of the first job on the second job;
- wherein the automatically inferring the first dependency is based at least in part on automatically determining that the source data object specified in the first job definition is the output data object specified in the second job definition;
- based at least in part on the automatically inferring the first dependency, dispatching, by the computer system, a first plurality of job instances for execution, wherein the dispatching the first plurality of job instances is controlled such that a first instance of the first job, of the first plurality of job instances, is blocked from being dispatched until after a first instance of the second job, of the first plurality of job instances, completes;
- after the dispatching the first plurality of job instances for execution, receiving, at the computer system, via graphical user interface controls of the job creation graphical user interface, a third job definition;
- wherein the third job definition is for a third job;
- wherein the third job definition specifies at least a source data object to provide input data to be consumed by the third job, and an output data object to store output data produced by the third job;
- based at least on the third job definition, automatically inferring, by the computer system, a second dependency;
- wherein the second dependency is the first job on the third job;
- wherein the second dependency is automatically inferred based at least in part on automatically determining that the source data object specified in the first job definition is the output data object specified in the third job definition;
- and
- based at least in part on the automatically inferring the first dependency and the automatically inferring the second dependency, dispatching, by the computer system, a second plurality of job instances for execution, wherein the dispatching the second plurality of job instances is controlled such that a second instance of the first job, of the second plurality of job instances, is blocked from being dispatched until after both:
- (a) a second instance of the second job, of the second plurality of job instances, completes, and (b) a first instance of the third job, of the second plurality of job instances, completes.
14 more elements
A computer system comprising:
- one or more processing units;
- memory;
- one or more programs stored in the memory and configured for execution by the one or more processing units, the one or more programs comprising instructions configured for:
- receiving, at the computer system, via graphical user interface controls of a job creation graphical user interface, a plurality of job definitions for a plurality of jobs, each job definition of the plurality of job definitions specifying at least a source data object to provide input data to be consumed by the job, and an output data object to store output data produced by the job;
- wherein the plurality of job definitions includes a first job definition and a second job definition;
- wherein the plurality of jobs includes a first job and a second job;
- wherein the first job definition is for the first job;
- wherein the second job definition is for the second job;
- automatically inferring, by the computer system, a first dependency of the first job on the second job;
- wherein the automatically inferring the first dependency is based at least in part on automatically determining that the source data object specified in the first job definition is the output data object specified in the second job definition;
- based at least in part on the automatically inferring the first dependency, dispatching, by the computer system, a first plurality of job instances for execution, wherein the dispatching the first plurality of job instances is controlled such that a first instance of the first job, of the first plurality of job instances, is blocked from being dispatched until after a first instance of the second job, of the first plurality of job instances, completes;
- after the dispatching the first plurality of job instances for execution, receiving, at the computer system, via graphical user interface controls of the job creation graphical user interface, a third job definition;
- wherein the third job definition is for a third job;
- wherein the third job definition specifies at least a source data object to provide input data to be consumed by the third job, and an output data object to store output data produced by the third job;
- based at least on the third job definition, automatically inferring, by the computer system, a second dependency;
- wherein the second dependency is the first job on the third job;
- wherein the second dependency is automatically inferred based at least in part on automatically determining that the source data object specified in the first job definition is the output data object specified in the third job definition;
- and
- based at least in part on the automatically inferring the first dependency and the automatically inferring the second dependency, dispatching, by the computer system, a second plurality of job instances for execution, wherein the dispatching the second plurality of job instances is controlled such that a second instance of the first job, of the second plurality of job instances, is blocked from being dispatched until after both:
- (a) a second instance of the second job, of the second plurality of job instances, completes, and (b) a first instance of the third job, of the second plurality of job instances, completes.
17 more elements
Dependencies are inferred from the claim text.