Message-based coordination of container-supported many task computing

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:

• receive, at the at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:
• the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;
• the job flow definition is stored among multiple job flow definitions within at least one federated area;
• the set of task routines is stored among multiple task routines within the at least one federated area;
• and
• the at least one federated area is maintained within at least one storage device;
• retrieve the job flow definition from among the multiple job flow definitions in the at least one federated area;
• store, within a job queue, a job performance request message comprising the job flow definition;
• within a first performance container, execute instructions of a first instance of a performance routine to cause the at least one processor to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
• derive an order of performance of the set of tasks that specifies at least a first task of the set of tasks to be performed and a second task to be performed after the first task, and that is based on indications in the job flow definition of dependencies among the set of tasks;
• store, within a task queue, a first task routine execution request message comprising an identifier associated with the first task;
• and
• in response to at least storage of an execution completion message indicative of completion of execution of the first task routine within the task queue, store, within the task queue, a second task routine execution request message comprising an identifier associated with the second task to cause a latest version of a second task routine to be executed within a second task container to cause the at least one processor to perform the second task;
• and
• within a first task container, in response to the storage of the first task routine execution request message within the task queue, perform operations comprising:
• use the identifier associated with the first task to retrieve a latest version of a first task routine of the set of task routines from the at least one federated area;
• commence execution, within the first task container, of instructions of the first task routine to commence performance of the first task;
• and
• following completion of the execution of the first task routine, store, within the task queue, the execution completion message indicative of completion of execution of the first task routine.

A computer-implemented method comprising:

• receiving, by at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:
• the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during the job flow performance;
• the job flow definition is stored among multiple job flow definitions within at least one federated area;
• the set of task routines is stored among multiple task routines within the at least one federated area;
• and
• the at least one federated area is maintained within at least one storage device;
• retrieving the job flow definition from among the multiple job flow definitions in the at least one federated area;
• storing, within a job queue, a job performance request message comprising the job flow definition;
• within a first performance container, executing, by the at least one processor, instructions of a first instance of a performance routine causing the at least one processor to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
• deriving, by the at least one processor, an order of performance of the set of tasks that specifies at least a first task of the set of tasks to be performed and a second task to be performed after the first task, and that is based on indications in the job flow definition of dependencies among the set of tasks;
• storing, within a task queue, a first task routine execution request message comprising an identifier associated with the first task;
• and
• in response to at least storage of an execution completion message indicative of completion of execution of the first task routine within the task queue, storing, within the task queue, a second task routine execution request message comprising an identifier associated with the second task to cause a latest version of a second task routine to be executed within a second task container to cause the at least one processor to perform the second task;
• and
• within a first task container, in response to the storage of the first task routine execution request message within the task queue, performing operations comprising:
• using the identifier associated with the first task to retrieve a latest version of a first task routine of the set of task routines from the at least one federated area;
• commencing execution of, by the at least one processor and within the first task container, instructions of the first task routine to commence performance of the first task;
• and
• following completion of the execution of the first task routine, storing, within the task queue, the execution completion message indicative of completion of execution of the first task routine.