Automated message-based job flow resource management in container-supported many task computing

A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause at least one processor to perform operations comprising:

• receive, at the at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:
• the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;
• the job flow definition is stored among multiple job flow definitions within at least one federated area;
• the set of task routines is stored among multiple task routines within the at least one federated area;
• and
• the at least one federated area is maintained within at least one storage device;
• retrieve the job flow definition from among the multiple job flow definitions in the at least one federated area;
• store, within a job queue, a job performance request message comprising the job flow definition;
• within a first performance container, execute instructions of a first instance of a performance routine to cause the at least one processor to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
• based on the data dependencies among the set of tasks, derive an order of performance of the set of tasks that specifies at least a first task of the set of tasks to be performed;
• store, within a task queue, a first task routine execution request message comprising an identifier associated with the first task;
• provide, to a resource allocation routine executed by the at least one processor, an indication of a need for provision of a first task container in which to execute a first task routine of the set of task routines to perform the first task, wherein execution of the resource allocation routine causes the at least one processor to dynamically allocate multiple containers based on availability of at least one of processing resources and storage resources;
• and
• in response to at least storage of an execution completion message within the task queue, store a job completion message indicative of completion of the job flow within the job queue;
• and
• in response to storage of the job completion message within the job queue, perform operations comprising:
• provide, to the resource allocation routine, an indication of cessation of the need for provision of the first task container;
• and
• transmit an indication of completion of the job flow to the requesting device via the network.

A computer-implemented method comprising:

• receiving, by at least one processor and from a requesting device via a network, a request to perform a job flow, wherein:
• the job flow is defined in a job flow definition that specifies a set of tasks to be performed via execution of a corresponding set of task routines during a performance of the job flow, and that specifies data dependencies among the set of tasks;
• the job flow definition is stored among multiple job flow definitions within at least one federated area;
• the set of task routines is stored among multiple task routines within the at least one federated area;
• and
• the at least one federated area is maintained within at least one storage device;
• retrieving the job flow definition from among the multiple job flow definitions in the at least one federated area;
• storing, within a job queue, a job performance request message comprising the job flow definition;
• within a first performance container, executing, by the at least one processor, instructions of a first instance of a performance routine to, in response to the storage of the job performance request message within the job queue, perform operations comprising:
• based on the data dependencies among the set of tasks, deriving, by the at least one processor, an order of performance of the set of tasks that specifies at least a first task of the set of tasks to be performed;
• storing, within a task queue, a first task routine execution request message comprising an identifier associated with the first task;
• providing, to a resource allocation routine executed by the at least one processor, an indication of a need for provision of a first task container in which to execute a first task routine of the set of task routines to perform the first task, wherein execution of the resource allocation routine, by the at least one processor, causes the at least one processor to dynamically allocate multiple containers based on availability of at least one of processing resources and storage resources;
• and
• in response to at least storage of an execution completion message within the task queue, storing a job completion message indicative of completion of the job flow within the job queue;
• and
• in response to storage of the job completion message within the job queue, performing operations comprising:
• providing, to the resource allocation routine, an indication of cessation of the need for provision of the first task container;
• and
• transmitting an indication of completion of the job flow to the requesting device via the network.