Cost-based optimization of configuration parameters and cluster sizing for hadoop

A method comprising:

• at a processor and memory:
• receiving at least one job profile of a MapReduce job;
• using the at least one job profile to predict execution of the MapReduce job within a predetermined setting of a distributed data processing system;
• and
• determining performance of the MapReduce job based on the prediction.

A method comprising:

• at a processor and memory:
• receiving at least one job profile of a MapReduce job;
• using the at least one job profile to predict execution of the MapReduce job within a plurality of different predetermined settings of a distributed data processing system;
• and
• determining one of the predetermined settings that optimizes performance of the MapReduce job.

A method comprising:

• at a processor and memory:
• receiving at least one job profile of a MapReduce job;
• and
• using the at least one job profile to determine one of the predetermined settings that optimizes performance of the MapReduce job within a plurality of different predetermined settings of a distributed data processing system

A computing device comprising:

• a processor and memory configured to:
• receive at least one measure of performance of a MapReduce job;
• determine a job profile based on the at least one measure of performance;
• and
• provide the job profile for one of user interface and prediction processes.

A computing device comprising:

• a processor and memory configured to:
• receive at least one job profile of a MapReduce job;
• use the at least one job profile to predict execution of the MapReduce job within a predetermined setting of a distributed data processing system;
• and
• determine performance of the MapReduce job based on the prediction.

A computing device comprising:

• a processor and memory configured to:
• receive at least one job profile of a MapReduce job;
• use the at least one job profile to predict execution of the MapReduce job within a plurality of different predetermined settings of a distributed data processing system;
• and
• determine one of the predetermined settings that optimizes performance of the MapReduce job.