System, method, and computer program product for processing large data sets by balancing entropy between distributed data segments

A load balancing system comprising at least one processor programmed or configured to:

• identify a data set stored in a storage array in communication with the at least one processor through a parallel data processing system, the data set comprising a plurality of records, each record of the plurality of records comprising at least one parameter value for at least one parameter;
• sort the data set based on the at least one parameter, resulting in sorted values;
• store the sorted values in memory;
• determine a distribution entropy of the data set based on the at least one parameter value of each record of the plurality of records;
• determine a number of segments based on the plurality of records;
• determine an average entropy of each segment of the number of segments by dividing the distribution entropy by the number of segments;
• segment the data set into the number of segments based on the average entropy, such that each segment comprises a segment entropy equal to the average entropy or within a tolerance of the average entropy, by storing each segment in a separate data structure of a plurality of data structures, each segment representing a range of values of the at least one parameter value and comprising one or more different records of the plurality of records within the range;
• and
• distribute processing tasks associated with each segment of the number of segments to a plurality of processors to process each record in each respective segment to thereby cause execution of the processing tasks by the plurality of processors.

A computer program product comprising at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor, cause the at least one processor to:

• identify a data set stored in a storage array in communication with the at least one processor through a parallel data processing system, the data set comprising a plurality of records, each record of the plurality of records comprising at least one parameter value for at least one parameter;
• sort the data set based on the at least one parameter, resulting in sorted values;
• store the sorted values in memory;
• determine a distribution entropy of the data set based on the at least one parameter value of each record of the plurality of records;
• determine a number of segments based on the plurality of records;
• determine an average entropy of each segment of the number of segments by dividing the distribution entropy by the number of segments;
• segment the data set into the number of segments based on the average entropy, such that each segment comprises a segment entropy equal to the average entropy or within a tolerance of the average entropy, by storing each segment in a separate data structure of a plurality of data structures, each segment representing a range of values of the at least one parameter value and comprising one or more different records of the plurality of records within the range;
• and
• distribute processing tasks associated with each segment of the number of segments to a plurality of processors to process each record in each respective segment to thereby cause execution of the processing tasks by the plurality of processors.