Cloud pipeline

A computer program product for establishing a cloud pipeline for a data processing job, the computer program product comprising:

• a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising:
• computer readable program code configured to design a plan for a cloud pipeline for performing the data processing job based, at least in part, on cloud service provider data about a set of one or more cloud service providers, wherein the plan comprises a set of processing stages each of which indicates processing upon a subset of a data set of the data processing job;
• computer readable program code configured to map allocated resources of the set of one or more cloud service providers to the plurality of processing stages;
• computer readable program code configured to generate instructions and software images based, at least in part, on the plan, wherein the instructions and the software images implement the cloud pipeline for performing the data processing job;
• computer readable program code configured to transmit the instructions and the software images to machines of the set of one or more cloud service providers, wherein the allocated resources comprise the machines;
• computer readable program code configured to monitor the machines and the data processing job;
• and
• computer readable program code configured to adapt the cloud pipeline to a failure if the computer readable program code configured to monitor the machines and the data processing job detects a failure.

An apparatus comprising:

• a network interface;
• a processor;
• and
• a cloud pipeline manager operable to,
• design a plan for a cloud pipeline for performing the data processing job based, at least in part, on cloud service provider data about a set of one or more cloud service providers, wherein the plan comprises a plurality of processing stages each of which indicates processing upon a subset of a data set of the data processing job;
• map allocated resources of the set of one or more cloud service providers to the plurality of processing stages;
• generate instructions and software images based, at least in part, on the plan, wherein the instructions and the software images implement the cloud pipeline for performing the data processing job;
• transmit the instructions and the software images to machines of the set of one or more cloud service providers, wherein the allocated resources comprise the machines;
• monitor the machines and the data processing job;
• and
• adapt the cloud pipeline to a failure if the computer readable program code configured to monitor the machines and the data processing job detects a failure.