System and method for efficiently converting low-locality data into high-locality data

A system for processing unstructured source data, the system comprising:

• an off-chip storage device;
• an on-chip storage device;
• and
• a processor coupled to the on-chip and off-chip storage devices, the processor being configured to:
• determine a target number of T partitions;
• identify the unstructured source data as input data for a first level of recursion;
• recursively generate partitions based on the input data until the target number of T partitions are generated, wherein the processor is configured to:
• load the input data from the off-chip storage device to the on-chip storage device, the input data having a range V;
• partition the input data into P temporary partitions via the on-chip storage device, wherein a particular one of the P temporary partitions has a range of V/P;
• store the P temporary partitions from the on-chip storage device to the off-chip storage device;
• and
• identify data in one of the P temporary partitions as the input data for a next level of recursion,
• wherein an application is configured to perform accesses of partitioned data from the T partitions for generating an output, wherein the accesses of the partitioned data are sequential read accesses of the off-chip storage device and wherein T, V, and P are numerical values greater than 1.