Systems and methods for generating performance prediction model and estimating execution time for applications

A processor implemented system comprising:

• a memory storing instructions;
• a communication interface;
• and
• one or more hardware processors coupled to said memory, wherein said one or more hardware processor (304) are configured by said instructions to:
• execute one or more synthetic benchmarks on a first dataset on a first cluster comprising a first set of nodes, wherein each of said one or more synthetic benchmarks comprises a MapReduce (MR) job, said MR job comprises a map task and a reduce task;
• extract one or more sensitive parameters related to performance of each sub-phase in said MR job, wherein said sensitive parameters are extracted based on a job execution log obtained upon executing of said MR job, and wherein said one or more sensitive parameters comprises an input size per map task, map selectivity, number of map tasks, number of reduce tasks, record selectivity, concurrent map tasks per node, one or more map waves, and disk resource contention delay;
• generate a linear regression prediction model using said one or more sensitive parameters to obtain a set of linear regression prediction models, and
• generate a performance prediction model based on said set of linear regression prediction models generated using said first dataset;
• wherein the linear regression prediction model indicates uniform growth in the data distribution, such that each phase of said MR job execution has proportionate increase in input data size with data growth;
• processing a Hive query such that said Hive query is translated into Directed Acyclic Graph (DAG) of said MR job;
• executing said MR job on a second dataset on a second cluster comprising a second set of nodes;
• extracting said one or more sensitive parameters from a job execution log obtained upon executing said DAG of said MR job, wherein the sensitive parameters are related to performance of the sub-phases of said MR job;
• and
• predicting, using said performance prediction model, a Hive query execution time of said DAG of said MR job based on the sensitive parameters from execution log obtained for the one or more sub-phases of said MR job executed on said second dataset.

One or more non-transitory machine readable information storage media storing instructions which, when executed by one or more hardware processors, causes the one or more hardware processors to perform a method comprising:

• executing, via said one or more hardware processors, one or more synthetic benchmarks on a first dataset on a first cluster comprising a first set of nodes, wherein each of said one or more synthetic benchmarks comprises a MapReduce (MR) job, said MR job comprises a map task and a reduce task;
• extracting, via said one or more hardware processors, one or more sensitive parameters related to performance of each sub-phase in said MR job, wherein said sensitive parameters are extracted based on a job execution log obtained upon executing of said MR job b, and wherein said one or more sensitive parameters comprises at least one of an input size per map task, map selectivity, number of map tasks, number of reduce tasks, record selectivity, concurrent map tasks per node, one or more map waves, and disk resource contention delay;
• generating, via said one or more hardware processors, a linear regression prediction model using said one or more sensitive parameters to obtain a set of linear regression prediction models;
• and
• generating, via said one or more hardware processors, a performance prediction model based on said set of linear regression prediction models generated using said first dataset;
• wherein the linear regression prediction model indicates uniform growth in the data distribution, such that each phase of said MR job execution has proportionate increase in input data size with data growth;
• processing, via said one or more hardware processors, a Hive query such that said Hive query is translated into Directed Acyclic Graph (DAG) of said MR job;
• executing, via said one or more hardware processors, said MR job on a second dataset on a second cluster comprising a second set of nodes;
• extracting, via said one or more hardware processors, said one or more sensitive parameters from a job execution log obtained upon executing said DAG of said MR job, wherein the sensitive parameters are related to performance of the sub-phases of said MR job;
• and
• predicting, using said performance prediction model, a Hive query execution time of said DAG of said MR job based on the sensitive parameters from execution log obtained for the one or more sub-phases of said MR job executed on said second dataset.