Detecting and Reordering Fixed-Length Records to Facilitate Compression

A computing device comprising:

• a processor;
• and
• programming instructions configured to be executed by the processor to detect a set of fixed-length records within a source file, including:
• determining a plurality of candidate records based on a repeating string pattern detected within the source file;
• determining a candidate record length based on a byte differential between repeated instances of the string pattern;
• comparing first byte values occurring between the repeated instances of the string pattern with second byte values following the first byte values by the record length in order to calculate a score;
• and
• if the score is less than or equal to a threshold value, determining that the candidate records are members of the set of fixed-length records.

An article of manufacture comprising:

• a storage medium;
• and
• a plurality of programming instructions stored on the storage medium and configured to program a computing device to, when executed:
• detect one or more fixed-length records within a source file, the detecting including
• determining a plurality of candidate records based on a repeating string pattern detected within the source file;
• determining a candidate record length based on a byte differential between repeated instances of the string pattern;
• comparing first byte values occurring between the repeated instances of the string pattern with second byte values following the first byte values by the record length in order to calculate a score;
• and
• if the score is less than or equal to a threshold value, determining that the candidate records are members of the set of fixed-length records;
• determine a plurality of column groupings within the fixed-length records, the determining including
• determining a dependency relationship between a first column and a second column based on a tendency of values repeating in the first column to appear together with values repeating in the second column, or
• determining a partition plan of a set of partition plans partitioning the fixed-length records into candidate column groupings as an optimal partition plan based on coding bit sizes associated with each partition plan;
• and
• reorder the fixed-length records based on the plurality of column groupings to facilitate compression of the source file.