Data processing pipeline with data regression framework

A method, comprising:

• receiving a request from a pipeline manager executing on a host platform to execute a job comprising a sequence of job steps;
• in response to the request, instantiating a sequence of job step executors within the data processing pipeline before execution of the job, wherein the instantiating comprises resolving dependencies of the job step executors in the sequence of job step executors;
• executing the sequence of job steps;
• recording current job execution data generated by the sequence of job step executors when the job is executed;
• comparing a current job step execution dataset to the historical job execution data in the data store;
• identifying a deviation in the sequence of job steps based on the comparison;
• and
• in response to the deviation being identified, generating a failure alert and capture metadata about a job step with the deviation from the pipeline manager;
• wherein the sequence of job steps is determined using a Directed Acyclic Graph (DAG) of executors generated based on the dependencies of the job step executors.

A non-transitory computer storage medium comprising instructions that, when executed by a processor, cause the processor to perform:

• receiving a request from a pipeline manager executing on a host platform to execute a job comprising a sequence of job steps;
• in response to the request, instantiating a sequence of job step executors within the data processing pipeline before execution of the job, wherein the instantiating comprises resolving dependencies of the job step executors in the sequence of job step executors;
• executing the sequence of job steps;
• recording current job execution data generated by the sequence of job step executors when the job is executed;
• comparing a current job step execution dataset to the historical job execution data in the data store;
• identifying a deviation in the sequence of job steps based on the comparison;
• and
• in response to the deviation being identified, generating a failure alert and capture metadata about a job step with the deviation from the pipeline manager;
• wherein the sequence of job steps is determined using a Directed Acyclic Graph (DAG) of executors generated based on the dependencies of the job step executors.