US 10,705,877 B2Granted

Workload automation and data lineage analysis

AssigneeAb Initio Technology LLC

InventorsHarry Michael Wolfson, Joel Gould, Anthony Yeracaris, Tim Wakeling

Life of this patent

Estimating the term…

Loading patent…

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for workload automation and job scheduling information. One of the methods includes obtaining job dependency information, the job dependency information specifying an order of execution of a plurality of jobs. The method also includes obtaining data lineage information that identifies dependency relationships between data stores and transformation, wherein at least one transformation accepts data from a first data store and produces data for a second data store. The method also includes creating links between the job dependency information and the data lineage information. The method also includes determining an impact of a change in a planned execution of an application of the plurality of applications based on the job dependency information, the created links, and the data lineage information.

Claims33

Dependent claims

A computer implemented method including:

obtaining, by a computer system, job dependency information that is indicative of an order of execution of a plurality of jobs, with at least some of the jobs including a reference to a first set of data store and data transform elements;
wherein obtaining the job dependency information includes:
obtaining scheduling information from a work load repository associated with a workload automation system;
and
identifying first data lineage information that identifies at least one dependency relationship between the first set of data store and/or data transform elements referenced by the at least some of the jobs and a second set of one or more data store and data transform elements, from the work load repository;
producing, by the computer system, one or more first links between jobs in the plurality of jobs, and one or more second links between the job dependency information that is indicative of the order of execution of the plurality of jobs and one or more identified elements of the first data lineage information;
receiving, by the computer system, a query that identifies a given job and a change to a planned execution of the given job;
and
for the change in the planned execution of the given job:
identifying, for the given job, one or more of the one or more produced first links that are between the given job and one or more remaining ones of the plurality of jobs and one or more of the one or more produced second links that are between the job dependency information that is indicative of the order of execution of the plurality of jobs and an identified element of the first data lineage information that is referenced by the job dependency information;
and
executing, by the computer system, an analysis of the identified links, the job dependency information and the first data lineage information to determine an impact of the change in the planned execution of the given job, and wherein determining the impact further includes:
identifying one or more further links between (i) additional scheduling information associated with one or more jobs or second data lineage information, and (ii) third data lineage information associated with one or more elements that are impacted by the given job;
and
identifying the impact further based on the one or more further links.

12 more elements

A non-transitory computer storage medium encoded with computer program instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

obtaining job dependency information that is indicative of an order of execution of a plurality of jobs, with at least some of the jobs including a reference to a first set of data store and data transform elements;
wherein obtaining the job dependency information includes:
obtaining scheduling information from a work load repository associated with a workload automation system;
and
identifying first data lineage information that identifies at least one dependency relationship between the first set of data store and data transform elements referenced by the at least some of the jobs and a second set of one or more data store and data transform elements, from the work load repository;
producing one or more first links between jobs in the plurality of jobs, and one or more second links between the job dependency information that is indicative of the order of execution of the plurality of jobs and one or more identified elements of the first data lineage information;
receiving a query that identifies a given job and a change to a planned execution of the given job;
and
for the change in the planned execution of the given job:
identifying, for the given job, one or more of the one or more produced first links that are between the given job and one or more remaining ones of the plurality of jobs and one or more of the one or more produced second links that are between the job dependency information that is indicative of the order of execution of the plurality of jobs and an identified element of the first data lineage information that is referenced by the job dependency information;
and
executing an analysis of the identified links, the job dependency information and the first data lineage information to determine an impact of the change in the planned execution of the given job, and wherein determining the impact further includes:
identifying one or more further links between (i) additional scheduling information associated with one or more jobs or second data lineage information, and (ii) third data lineage information associated with one or more elements that are impacted by the given job;
and
identifying the impact further based on the one or more further links.

12 more elements

A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
obtaining job dependency information that is indicative of an order of execution of a plurality of jobs, with at least some of the jobs including a reference to a first set of data store and data transform elements;
wherein obtaining the job dependency information includes:
obtaining scheduling information from a work load repository associated with a workload automation system;
and
identifying first data lineage information that identifies at least one dependency relationship between the first set of data store and data transform elements referenced by the at least some of the jobs and a second set of one or more data store and/or data transform elements, from the work load repository;
producing one or more first links between jobs in the plurality of jobs, and one or more second links between the job dependency information that is indicative of the order of execution of the plurality of jobs and one or more identified elements of the first data lineage information;
receiving a query that identifies a given job and a change to a planned execution of the given job;
and
for the change in the planned execution of the given job:
identifying, for the given job, one or more of the one or more produced first links that are between the given job and one or more remaining ones of the plurality of jobs and one or more of the one or more produced second links that are between the job dependency information that is indicative of the order of execution of the plurality of jobs and an identified element of the first data lineage information that is referenced by the job dependency information;
and
executing an analysis of the identified links, the job dependency information and the first data lineage information to determine an impact of the change in the planned execution of the given job, and wherein determining the impact further includes:
identifying one or more further links between (i) additional scheduling information associated with one or more jobs or second data lineage information, and (ii) third data lineage information associated with one or more elements that are impacted by the given job;
and
identifying the impact further based on the one or more further links.

13 more elements

Dependencies are inferred from the claim text.