Hybrid parallelization strategies for machine learning programs on top of MapReduce

A computer program product for parallelization, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

• receiving program code containing at least one parallel for statement having a plurality of iterations;
• determining a parallel execution plan for the program code, wherein determining the parallel execution plan comprises applying an heuristic optimizer, the heuristic optimizer comprises a cost model, and wherein the cost model includes at least one of an execution time of the program code and a memory estimate of the program code;
• according to the parallel execution plan, partitioning the plurality of iterations into a plurality of tasks, each task comprising at least one iteration, the iterations of each task being independent;
• determining first data required by the plurality of tasks;
• determining an access pattern by the plurality of tasks of the first data;
• based on the access pattern, partitioning the first data;
• providing each of the plurality of tasks to one of a plurality of parallel workers, each of the plurality of parallel workers being a MapReduce task or a local thread, wherein providing each of the plurality of tasks to one of the plurality of parallel workers comprises:
• determining second data required by a first of the plurality of tasks;
• determining that the second data is locally available to a first of the
• plurality of parallel workers;
• and
• providing the first of the plurality of tasks to the first of the plurality of parallel workers;
• receiving from each of the plurality of parallel workers a result;
• and
• aggregating the results from each of the plurality of parallel workers.