Orchestration for data pipeline execution plans

An apparatus for data processing, comprising:

• a processor;
• memory coupled with the processor;
• and
• instructions stored in the memory and executable by the processor to cause the apparatus to:
• receive, from a user device, a request defining a data modification process;
• generate an execution file based at least in part on the request, wherein the execution file comprises a set of tasks for performing the data modification process and an order for performing the set of tasks;
• execute, for the execution file, a first subset of tasks of the set of tasks according to the order for performing the set of tasks;
• update the execution file based at least in part on executing the first subset of tasks, wherein the updating comprises modifying a second subset of tasks of the set of tasks;
• and
• execute, for the updated execution file, the modified second subset of tasks according to the order for performing the set of tasks.

A non-transitory computer-readable medium storing code for data processing, the code comprising instructions executable by a processor to:

• receive, from a user device, a request defining a data modification process;
• generate an execution file based at least in part on the request, wherein the execution file comprises a set of tasks for performing the data modification process and an order for performing the set of tasks;
• execute, for the execution file, a first subset of tasks of the set of tasks according to the order for performing the set of tasks;
• update the execution file based at least in part on executing the first subset of tasks, wherein the updating comprises modifying a second subset of tasks of the set of tasks;
• and
• execute, for the updated execution file, the modified second subset of tasks according to the order for performing the set of tasks.