Size-based record partitioning

An apparatus, comprising:

• a processor;
• memory coupled with the processor;
• and
• instructions stored in the memory and executable by the processor to cause the apparatus to:
• receive a plurality of records representing changes made to a plurality of datasets, the plurality of records comprising a plurality of data change instances associated with a plurality of data schema types, wherein the plurality of data change instances corresponds to a plurality of respective keys;
• compute an average size of data for each data schema type of the plurality of data schema types;
• modify the plurality of respective keys through a salting process, wherein the salting process comprises assigning non-overlapping numerical ranges to subsets of the plurality of respective keys;
• sort, using the non-overlapping numerical ranges, the plurality of data change instances into a plurality of partitions of a target file size;
• and
• write files of the target file size to a data store.

A non-transitory computer-readable medium storing code, the code comprising instructions executable by a processor to:

• receive a plurality of records representing changes made to a plurality of datasets, the plurality of records comprising a plurality of data change instances associated with a plurality of data schema types, wherein the plurality of data change instances corresponds to a plurality of respective keys;
• compute an average size of data for each data schema type of the plurality of data schema types;
• modify the plurality of respective keys through a salting process, wherein the salting process comprises assigning non-overlapping numerical ranges to subsets of the plurality of respective keys;
• sort, using the non-overlapping numerical ranges, the plurality of data change instances into a plurality of partitions of a target file size;
• and
• write files of the target file size to a data store.