A system for optimizing a parameter configuration of a distributed computing job, comprising a processor and a memory storing program codes, wherein the processor performs the stored program codes for:
- obtaining job programs of different distributed computing jobs, and determining a key parameter configuration set based on the job programs of the different distributed computing jobs, wherein the key parameter configuration set comprises multiple key parameter configuration item combinations and an execution time corresponding to each of the key parameter configuration item combinations, each of the key parameter configuration item combinations comprises configuration items of multiple key parameters, configuration item of each key parameter is randomly generated within a value range of the key parameter, and each of the key parameters is a parameter whose effect degree on an execution time of each of the distributed computing jobs is higher than an effect degree threshold;
- obtaining a cluster status during execution of each of the distributed computing jobs, randomly generating a sample data set based on the key parameter configuration set and the cluster status, and establishing a performance prediction model by inputting a key parameter configuration item combination comprised in the sample data set and the cluster status and by outputting the execution time corresponding to each of the key parameter configuration item combinations, wherein the performance prediction model is configured to determine a key parameter configuration item combination with a shortest execution time, and the cluster status is a load size of a cluster;
- searching for an optimal configuration in a current cluster status based on the performance prediction model by using a multi-objective genetic algorithm and an optimization module configured with an optimal configuration selection strategy, wherein the optimization module is configured to simultaneously optimize the execution time and an amount of memory resource usage, and the optimal configuration is a configuration with shortest execution time;
- obtaining a job program of a to-be-optimized distributed computing job and a cluster status during execution of the to-be-optimized distributed computing job, and determining a to-be-optimized key parameter configuration item combination based on the job program of the to-be-optimized distributed computing job;
- and
- inputting, to the performance prediction model, the to-be-optimized key parameter configuration item combination and the cluster status during execution of the to-be-optimized distributed computing job, and outputting a key parameter configuration item combination with a shortest execution time of the to-be-optimized distributed computing job, wherein a key parameter configuration of each key parameter configuration item in the key parameter configuration item combination with a shortest execution time of the to-be-optimized distributed computing job is an optimal configuration of the to-be-optimized distributed computing job;
- wherein the inputting, to the performance prediction model, the to-be-optimized key parameter configuration item combination and the cluster status during execution of the to-be-optimized distributed computing job, and outputting a key parameter configuration item combination with a shortest execution time of the to-be-optimized distributed computing job, comprises:
- determining an optimization result by inputting the to-be-optimized key parameter configuration item combination and the sampled cluster status during execution of the to-be-optimized distributed computing job to the performance prediction model and by optimizing the execution time of the to-be-optimized distributed computing job based on an NSGA-II algorithm with an objective of the shortest execution time and the least amount of memory resource usage;
- determining whether the optimization result meets a convergence condition, so as to obtain a first determining result, wherein the convergence condition is that searching stops if an optimal configuration obtained by searching based on the genetic algorithm does not change;
- using the key parameter configuration item combination with a shortest execution time as an optimal solution when the first determining result indicates that the optimization result meets the convergence condition;
- determining whether a configuration in which loss of performance is less than a maximum performance loss (MaxPL) and memory saving is larger than a minimum memory gain (MinMG) exists in the optimal solution, so as to obtain a second determining result;
- determining a key parameter configuration item combination:
- if the second determining result indicates that the optimal configuration exists in the optimal solution, obtain a key parameter configuration item combination with the least amount of memory resource usage by discarding a key parameter configuration item with a performance level lower than a performance level threshold;
- and
- determining a key parameter configuration item combination:
- if the second determining result indicates that the optimal configuration does not exist in the optimal solution, output the key parameter configuration item combination with a shortest execution time of the to-be-optimized distributed computing job.