Dataflow Execution Time Estimation for In-Memory Distributed Processing Framework

A system, comprising:

• a memory;
• and
• at least one processing device, coupled to the memory, operative to implement the following steps:
• obtaining an input dataset for a dataflow for execution in a distributed processing framework;
• determining a data unit for a given operation of the dataflow, wherein the data unit comprises a substantially minimal unit of data processed by the given operation of the dataflow;
• estimating a number of rounds required to execute a number of data units in the input dataset using one or more nodes of the distributed processing framework assigned to execute the given operation;
• determining, using the at least one processing device, an execution time spent by the given operation to process one data unit;
• estimating, using the at least one processing device, the execution time for the given operation based on the execution time spent by the given operation to process one data unit and the number of rounds required to execute the number of data units in the input dataset;
• and
• executing, using the at least one processing device, the given operation with the input dataset in the distributed processing framework.

A computer program product, comprising a tangible machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:

• obtaining an input dataset for a dataflow for execution in a distributed processing framework;
• determining a data unit for a given operation of the dataflow, wherein the data unit comprises a substantially minimal unit of data processed by the given operation of the dataflow;
• estimating a number of rounds required to execute a number of data units in the input dataset using one or more nodes of the distributed processing framework assigned to execute the given operation;
• determining, using the at least one processing device, an execution time spent by the given operation to process one data unit;
• estimating, using the at least one processing device, the execution time for the given operation based on the execution time spent by the given operation to process one data unit and the number of rounds required to execute the number of data units in the input dataset;
• and
• executing, using the at least one processing device, the given operation with the input dataset in the distributed processing framework.