Load balancing for parallel tasks

A method of distributing a plurality of computational workloads between a plurality of processing threads executing in a parallel computing system, the method comprising:

• arranging, by a processing node in the parallel computing system, the plurality of computational workloads in order of computational cost;
• dividing the arranged computational workloads into a plurality of work cycles, each work cycle having a number of arranged computational workloads equal to a multiple of the number of the plurality of processing threads;
• and
• distributing the computational workloads among the plurality of processing threads according to a sequence of distribution patterns, wherein:
• a first work cycle distributes its plurality of computational workloads to the plurality of processing threads according to a first distribution pattern in the sequence, and
• each subsequent work cycle distributes its plurality of computational workloads to the plurality of processing threads according to a distribution pattern that is the reverse of the distribution pattern immediately preceding it in the sequence,
• wherein a processing thread that receives a high cost workload in a given work cycle receives a low cost workload in a subsequent work cycle, thereby balancing the plurality of computational workloads among the plurality of processing threads.