System and Method for Optimizing the Evaluation of Task Dependency Graphs

A computer readable medium storing instructions that, when executed by a processor, cause the processor to optimized a task graph that delineates a plurality of tasks to be evaluated in a parallel processing environment, by performing the steps of:

• generating a first aggregation topology associated with the task graph that divides the plurality of tasks into a first collection of sets, wherein each set in the first collection of sets includes one or more tasks from the plurality of tasks, and each task of the plurality of tasks belongs to only one set included in the first collection of sets;
• compiling the plurality of tasks according to the first task aggregation topology to generate units of work to be executed in the parallel processing environment;
• collecting statistics associated with executing the units of work in the parallel processing environment;
• and
• determining whether the first task aggregation topology is more efficient in execution than any previously-defined task aggregation topology based on the statistics;
• and
• if the task aggregation topology is more efficient in execution than any previously-defined task aggregation topology, then selecting the first task aggregation topology as the most optimal task aggregation topology, or
• if the first task aggregation topology is not more efficient in execution than any previously-defined task aggregation topology, then selecting a second task aggregation topology as the most optimal task aggregation topology.

A computer system, comprising:

• a parallel processing environment having one or more processing engines;
• and
• an optimization engine configured to optimize a task graph delineates a plurality of tasks to be evaluated in the parallel processing environment, by performing the steps of:
• generating a first aggregation topology associated with the task graph that divides the plurality of tasks into a first collection of sets, wherein each set in the first collection of sets includes one or more tasks from the plurality of tasks, and each task of the plurality of tasks belongs to only one set included in the first collection of sets;
• compiling the plurality of tasks according to the first task aggregation topology to generate units of work to be executed in the parallel processing environment;
• collecting statistics associated with executing the units of work in the parallel processing environment;
• and
• determining whether the first task aggregation topology is more optimal than any previously-defined task aggregation topology based on the statistics;
• and
• if the task aggregation topology is more optimal than any previously-defined task aggregation topology, then selecting the first task aggregation topology as the most optimal task aggregation topology, or
• if the first task aggregation topology is not more optimal than any previously-defined task aggregation topology, then selecting a second task aggregation topology as the most optimal task aggregation topology.