Method and apparatus for estimating a completion time for mapreduce jobs

A non-transitory computer-readable medium storing a plurality of instructions which, when executed by a processor, cause the processor to perform operations for estimating a completion time for a MapReduce job, the operations comprising:

• building a general MapReduce performance model;
• computing one or more performance characteristics of each one of one or more benchmark workloads in accordance with the general MapReduce performance model in a known processing system;
• computing one or more performance characteristics of the MapReduce job in accordance with the general MapReduce performance model in the known processing system;
• selecting a subset of the one or more benchmark workloads that have similar performance characteristics as the one or more performance characteristics of the MapReduce job;
• targeting a cluster of processing nodes in a distributed processing system having one or more unknown hardware configurations;
• computing one or more performance characteristics of the subset of the one or more benchmark workloads in the cluster of processing nodes;
• and
• estimating the completion time for the MapReduce job based upon a comparative analysis of the one or more performance characteristics of the subset of the one or more benchmark workloads in the cluster of processing nodes and the one or more performance characteristics of the subset of the one or more benchmark workloads in the known processing system.

A method for estimating a completion time for a MapReduce job, comprising:

• building, by a processor, a general MapReduce performance model that is represented by a first function,
• Total Time=Map Phase Duration+Shuffle Phase Duration+Reduce Phase Duration−Overlap Duration,
• wherein the function may be represented by a vector comprising each component of the Map Phase Duration, the Shuffle Phase Duration, the Reduce Phase duration and the Overlap Duration, the vector represented by an expression,
• V=[FSreadmap,FSwritemap,HDFSreadmap,HDFSwritemap,CPUmap,ShuffleTime,FSwriteshuffle,FSreadreduce,FSwritereduce,HDFSreadreduce,HDFSwritereduce,CPUreduce,OverlapTime];
• computing, by the processor, a time value for each component of the vector for each one of one or more benchmark workloads in a known processing system;
• computing, by the processor, a time value for each component of the vector for the MapReduce job in the known processing system;
• calculating, by the processor, a distance between the time value of the each component of the vector for each one of the one or more benchmark workloads and the time value of each component of the vector for the MapReduce job;
• selecting, by the processor, a subset of the one or more benchmark workloads that have similar performance characteristics as the MapReduce job based upon the subset of the one or more benchmark workloads having the most components with the distance below a threshold;
• targeting, by the processor, a cluster of processing nodes in a distributed processing system having one or more unknown hardware configurations;
• computing, by the processor, a time value of the each component of the vector of each one of the subset of the one or more benchmark workloads in the cluster of processing nodes;
• and
• estimating, by the processor, the completion time for the MapReduce job based upon the time value of the each component of the vector of the each one of the subset of the one or more benchmark workloads, wherein the completion time is estimated using a second function,
• Estimated
• 
• 
• Completion
• 
• 
• Time
• =
• ∑
• j
• 
• (
• V
• unknown
• ·
• C
• j
• /
• V
• known
• ·
• C
• j
• )
• 
• V
• job
• known
• ·
• C
• j
• ,
• wherein j is a jth component of the vector, Vunknown·Cj is an average time value of the each component of a respective vector of each one of the subset of the one or more benchmark workloads that is measured in the cluster of the distributed processing system, Vknown·Cj is an average time value of the each component of the respective vector for the each one of the subset of the one or more benchmark workloads that is measured in the known processing system and Vjobknown·Cj is an average time value of the each component of the vector for the MapReduce job measured in the known processing system.