Resilient scheduling of broker jobs for asynchronous tasks in a multi-tenant platform-as-a-service (PaaS) system

A system, comprising:

• a memory;
• and
• a processing device communicably coupled to the memory, the processing device to:
• receive, from a user device of a multi-tenant Platform-as-a-Service (PaaS) system, a first request to complete a job;
• collect information of a plurality of nodes, applications residing on the plurality of nodes, and software components utilized by the applications residing on the plurality of nodes;
• generate a model of a broker, wherein the model represents the plurality of nodes, the applications, and the software components, wherein the model specifies a plurality of sub-operations for the job and corresponding re-execution parameters for retrying or rolling back each of the sub-operations;
• invoke the model of the broker of the multi-tenant PaaS system to elaborate the job into the plurality of sub-operations;
• store the plurality of sub-operations to a data store of the broker, the plurality of sub-operations corresponding to the job in the data store;
• execute each sub-operation of the plurality of sub-operations by a worker component as part of processing of the job;
• complete the job when all of the plurality of the sub-operations are executed completely;
• and
• for each respective sub-operation of the plurality of sub-operations that does not execute completely, process the respective sub-operation according to the corresponding re-execution parameters corresponding to the respective sub-operation, wherein first re-execution parameters of a first operation of the plurality of sub-operations comprise an indication that the first operation is to re-execute as-is without regard to a state of a previous execution attempt, wherein second re-execution parameters of a second operation of the plurality of sub-operations specify a list of earlier operations that are to be rolled back before retry along with subsequent operations that depend on the earlier operations in the list.

A non-transitory machine-readable storage medium including instructions that, when accessed by a processing device, cause the processing device to:

• receive, by the processing device of a broker of a multi-tenant Platform-as-a-Service (PaaS) system from a user device of the multi-tenant PaaS system, a first request to complete a job;
• collect information of a plurality of nodes, applications residing on the plurality of nodes, and software components utilized by the applications residing on the plurality of nodes;
• generate a model of the broker using the information, wherein the model represents the plurality of nodes, the applications, and the software components, wherein the model specifies a plurality of sub-operations for the job and corresponding re-execution parameters for retrying or rolling back each of the sub-operations;
• invoke, by a worker component of a server device of the broker, the model of the broker to elaborate the job into the plurality of sub-operations, store the plurality of sub-operations in a data store, and associate the plurality of sub-operations with the job;
• and
• executing, by the worker component as part of processing the job, each of the plurality of sub-operations transactionally, wherein first re-execution parameters of a first operation of the plurality of sub-operations comprise an indication that the first operation is to re-execute as-is without regard to a state of a previous execution attempt, wherein second re-execution parameters of a second operation of the plurality of sub-operations specify a list of earlier operations that are to be rolled back before retry along with subsequent operations that depend on the earlier operations in the list.