Data processing pipeline failure recovery

A system comprising:

• one or more processors;
• and
• memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
• receiving a compute graph associated with a data processing pipeline, the compute graph identifying one or more components of the data processing pipeline and dependencies therebetween;
• causing the one or more components to execute;
• storing, based at least in part on receiving a first exit code in association with execution of a first component, a first indication that execution of the first component of the one or more components executed successfully;
• storing, in association with the first component and based at least in part on the execution of the first component, an output determined by the first component as a result of execution of the first component;
• storing, based at least in part on receiving a second exit code in association with execution of a second component, a second indication that the second component of the one or more components failed to execute successfully;
• retrieving, as a retrieved output, the output;
• and
• causing the second component to re-execute, as a re-execution, based at least in part on the retrieved output, wherein the re-execution starts at the second component and wherein re-executing the second component comprises providing the output determined by the first component as input to the second component.

A non-transitory computer-readable medium comprising processor-executable instructions that, when executed by one or more processors, cause the processors perform operations comprising:

• receiving a compute graph associated with a job, the compute graph identifying two or more components of the job and dependencies therebetween;
• causing the two or more components to execute;
• storing, based at least in part on receiving a first exit code in association with execution of a first component, a first indication that the execution of the first component of the two or more components executed successfully;
• storing, based at least in part on the execution of the first component, an output determined by the first component as a result of execution of the first component;
• storing, based at least in part on receiving a second exit code in association with execution of a second component, a second indication that the second component of the two or more components failed to execute successfully;
• and
• causing the second component to re-execute, based at least in part on the second indication, wherein the re-execution starts at the second component and wherein re-executing the second component comprises providing the output determined by the first component as input to the second component.