System and method for data pipeline optimization in an analytic applications environment

A method for use with an analytic applications environment, for controlling an order of running jobs associated with a data pipeline process and flow of data, comprising:

• providing, at a computer including one or more processors, access by an analytic applications environment to a data warehouse for storage of data by a plurality of tenants;
• wherein a data pipeline or other processing component performs at least one of an extract, transform, load or other data pipeline process to extract data from an enterprise application or database environment, to be loaded into a data warehouse;
• wherein each tenant can be associated with an activation plan that is used by the system to control operation of the data pipeline or process, or services, to address requirements of that tenant;
• wherein over a period of time, the system collects or records historical performance data associated with a plurality of activation plans for one or more tenants;
• and
• wherein based on a determination of historical performance data recorded over a period of time, the system can determine a priority or ordering of jobs, or otherwise optimize the execution of activation plans associated with a particular tenant, or across a sequence of activation plans associated with multiple tenants.

A non-transitory computer readable storage medium having instructions thereon, which when read and executed by a computer including one or more processors cause the computer to perform a method, comprising:

• providing, at a computer including one or more processors, access by an analytic applications environment to a data warehouse for storage of data by a plurality of tenants;
• wherein a data pipeline or other processing component performs at least one of an extract, transform, load or other data pipeline process to extract data from an enterprise application or database environment, to be loaded into a data warehouse;
• wherein each tenant can be associated with an activation plan that is used by the system to control operation of the data pipeline or process, or services, to address requirements of that tenant;
• wherein over a period of time, the system collects or records historical performance data associated with a plurality of activation plans for one or more tenants;
• and
• wherein based on a determination of historical performance data recorded over a period of time, the system can determine a priority or ordering of jobs, or otherwise optimize the execution of activation plans associated with a particular tenant, or across a sequence of activation plans associated with multiple tenants.