Dynamically Adapting Task Execution Parallelism of Distributed Applications

A computer system for scheduling tasks for concurrent execution in parallel, the computer system comprising:

• a communication fabric;
• a storage device connected to the communication fabric, wherein the storage device stores program instructions;
• and
• a processor connected to the communication fabric, wherein the processor executes the program instructions to:
• receive, from a set of worker nodes via a network, metrics corresponding to execution of tasks in a plurality of stages of a distributed application that are running on the set of worker nodes;
• perform, using a prediction model, an analysis of the metrics corresponding to the execution of the tasks in the plurality of stages of the distributed application received from the set of worker nodes;
• determine, using the prediction model, an optimal per-stage task execution parallelism level for each respective stage of the plurality of stages based on the analysis of the metrics corresponding to the execution of the tasks in the plurality of stages of the distributed application;
• and
• schedule the tasks to execute on the set of worker nodes based on the optimal per-stage task execution parallelism level determined for each respective stage of the plurality of stages to improve at least one of runtime of the distributed application and resource utilization by the distributed application.

A computer program product for scheduling tasks for concurrent execution in parallel, the computer program product comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:

• receive, from a set of worker nodes via a network, metrics corresponding to execution of tasks in a plurality of stages of a distributed application that are running on the set of worker nodes;
• perform, using a prediction model, an analysis of the metrics corresponding to the execution of the tasks in the plurality of stages of the distributed application received from the set of worker nodes;
• determine, using the prediction model, an optimal per-stage task execution parallelism level for each respective stage of the plurality of stages based on the analysis of the metrics corresponding to the execution of the tasks in the plurality of stages of the distributed application;
• and
• schedule the tasks to execute on the set of worker nodes based on the optimal per-stage task execution parallelism level determined for each respective stage of the plurality of stages to improve at least one of runtime of the distributed application and resource utilization by the distributed application.