Executing multi-stage distributed computing operations with independent rollback workflow

A non-transitory computer-readable storage medium containing program instructions for a method for executing multi-stage distributed computing operations, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform steps comprising:

• receiving a request to execute a multi-stage distributed computing operation;
• initiating an operation workflow for the multi-stage distributed computing operation in response to the received request, the operation workflow including tasks of the multi-stage distributed computing operation;
• executing the tasks of the operation workflow for the multi-stage distributed computing operation by a plurality of service compute nodes;
• when the last task of the operation workflow has been executed, modifying a message in a task queue to indicate that all the tasks of the operation workflow for the multi-stage distributed computing operation have been completed and enqueuing the modified message in the task queue;
• monitoring the tasks of the operation workflow for the multi-stage distributed computing operation being executed by the plurality of service compute nodes to detect any failures;
• initiating a rollback workflow for the multi-stage distributed computing operation when a failure of the operation workflow for the multi-stage distributed computing operation is detected, the rollback workflow including rollback tasks for the multi-stage distributed computing operation that correspond to the tasks of the operation workflow to roll back executed tasks of the operation workflow, wherein the rollback workflow is an independent workflow from the operation workflow;
• and
• executing the rollback tasks of the rollback workflow for the multi-stage distributed computing operation by the plurality of service compute nodes.

A system comprising:

• memory;
• and
• at least one processor configured to:
• receive a request to execute a multi-stage distributed computing operation;
• initiate an operation workflow for the multi-stage distributed computing operation in response to the received request, the operation workflow including tasks of the multi-stage distributed computing operation;
• execute the tasks of the operation workflow for the multi-stage distributed computing operation by a plurality of service compute nodes;
• when the last task of the operation workflow has been executed, modify a message in a task queue to indicate that all the tasks of the operation workflow for the multi-stage distributed computing operation have been completed and enqueue the modified message in the task queue;
• monitor the tasks of the operation workflow for the multi-stage distributed computing operation being executed by the plurality of service compute nodes to detect any failures;
• initiate a rollback workflow for the multi-stage distributed computing operation when a failure of the operation workflow for the multi-stage distributed computing operation is detected, the rollback workflow including rollback tasks for the multi-stage distributed computing operation that correspond to the tasks of the operation workflow to roll back executed tasks of the operation workflow, wherein the rollback workflow is an independent workflow from the operation workflow;
• and
• execute the rollback tasks of the rollback workflow for the multi- stage distributed computing operation by the plurality of service compute nodes.