Data compression system and method

A method of compressing a data file comprising a sequence of bytes of a length greater than or equal to a predefined length, the method including the steps of:

• retrieving the data file from a secondary storage device;
• storing the data file in direct access memory;
• calculating the frequency of unique byte values within a sub-sequence of the data file, the sub-sequence having a length not exceeding the predefined length;
• calculating the position of the one or more unique byte values within the sub-sequence;
• creating an index for the sub-sequence, the index including a data value representing the calculated frequency of unique byte values within the sub-sequence;
• and
• on the sub-sequence having a frequency of unique byte values below a predetermined threshold, applying a data transformation to the sub-sequence to increase the frequency of unique byte values in the sub-sequence and adding to the index a data value representing the data transformation;
• on the sub-sequence having a frequency of unique byte values above a predefined threshold, adding to the index a data value representing the position of one or more unique byte values within the sub-sequence;
• creating an output data file, the output data file having a file type identifier;
• and
• adding the index to the output data file.

A system for compressing a data file comprising a sequence of bytes of a length greater than or equal to a predefined length, where the system is configured to:

• retrieve the data file from a secondary storage device;
• store the data file in direct access memory;
• calculate the frequency of unique byte values within a sub-sequence of the data file, the sub-sequence having a length not exceeding the predefined length;
• create an index for the sub-sequence, the index including a data value representing the calculated frequency of unique byte values within the sub-sequence;
• and
• on the sub-sequence having a frequency of unique byte values below a predetermined threshold, apply a data transformation to the sub-sequence to increase the frequency of unique byte values in the sub-sequence and add to the index a data value representing the data transformation;
• on the sub-sequence having a frequency of unique byte values above a predefined threshold, add to the index a data value representing the position of one or more unique byte values within the sub-sequence;
• create an output data file, the output data file having a file type identifier;
• and
• add the index to the output data file.

A system for compressing a data file comprising a sequence of bytes of a length greater than or equal to a predefined length, where the system is configured to:

• retrieve the data file from a secondary storage device;
• store the data file in direct access memory;
• calculate the frequency of unique byte values within a sub-sequence of the data file, the sub-sequence having a length not exceeding the predefined length;
• and
• calculate the position of the one or more unique byte values within the sub-sequence;
• create an index for the sub-sequence, the index including a data value representing the calculated frequency of unique byte values within the sub-sequence;
• and
• on the sub-sequence having a frequency of unique byte values below a predetermined threshold, apply a data transformation to the sub-sequence to increase the frequency of unique byte values in the sub-sequence and add to the index a data value representing the data transformation;
• on the sub-sequence having a frequency of unique byte values above a predefined threshold, add to the index a data value representing the position of one or more unique byte values within the sub-sequence;
• create an output data file, the outout data file having a file type identifier;
• and add the index to the output data file.

A computer program stored on tangible storage medium comprising executable instructions for performing a method of compressing a data file comprising a sequence of bytes of a length greater than or equal to a predefined length, the method comprising:

• retrieving the data file from a secondary storage device;
• storing the data file in direct access memory;
• calculating the frequency of unique byte values within a sub-sequence of the data file, the sub-sequence having a length not exceeding the predefined length;
• creating an index for the sub-sequence, the index including a data value representing the calculated frequency of unique byte values within the sub-sequence;
• and
• on the sub-sequence having a frequency of unique byte values below a predetermined threshold, applying a data transformation to the sub-sequence to increase the frequency of unique byte values in the sub-sequence and adding to the index a data value representing the data transformation;
• on the sub-sequence having a frequency of unique byte values above a predefined threshold, adding to the index a data value representing the position of one or more unique byte values within the sub-sequence;
• creating an output data file, the output data file having a file type identifier;
• and
• adding the index to the output data file.

A computer program stored on tangible storage medium comprising executable instructions for performing a method of compressing a data file comprising a sequence of bytes of a length greater than or equal to a predefined length, the method comprising:

• retrieving the data file from a secondary storage device;
• storing the data file in direct access memory;
• calculating the frequency of unique byte values within a sub-sequence of the data file, the sub-sequence having a length not exceeding the predefined length;
• and
• calculating the position of the one or more unique values within the sub-sequence;
• creating an index for the sub-sequence, the index including a data value representing the calculated frequency of unique byte values within the sub-sequence;
• and
• on the sub-sequence having a frequency of unique byte values below a predetermined threshold, applying a data transformation to the sub-sequence to increase the frequency of unique byte values in the sub-sequence and adding to the index a data value representing the data transformation;
• on the sub-sequence having a frequency of unique byte values above a predefined threshold, adding to the index a data value representing the position of one or more unique byte values within the sub-sequence;
• creating an output data file, the output data file having a file type identifier;
• and
• adding the index to the output data file.

A method of modifying a data file prior to compression, comprising the steps of:

• (a) reading the data file into a sequence of multiple-byte data groups;
• (b) arranging the bytes of each data group in ascending byte value order;
• (c) writing an index into a first file that represents the original position of the bytes in each data group relative to their rearranged order;
• (d) writing the first byte value from each of the rearranged data groups into a second file;
• (e) writing calculated difference byte values representing the respective differences between each pair of consecutive byte values in each rearranged data group into a third file;
• and
• (f) outputting the first, second and third files for compression.

A method of modifying a data file prior to compression, the data file having a sequence of byte values and each byte value having an associated decimal order value representing its position in the sequence of the byte values of the data file, the method comprising the steps of:

• (a) writing the byte values having an even order value into a first file;
• (b) writing the byte values having an odd order value into a second file;
• and
• (c) outputting the first and second files to a lossless compression algorithm for compression into a single file.

A method for decompressing a data file, comprising:

• receiving a data file comprising a compressed original data file, the original data file comprising a sequence of bytes of a length greater than or equal to a predefined length and having been compressed by a compression process comprising:
• retrieving the original data file from a secondary storage device;
• storing the original data file in direct access memory;
• calculating the frequency of unique byte values within a sub-sequence of the original data file, the sub-sequence having a length not exceeding the predefined length;
• creating an index for the sub-sequence, the index including a data value representing the calculated frequency of unique byte values within the sub-sequence;
• on the sub-sequence having a frequency of unique byte values below a predetermined threshold, applying a data transformation to the sub-sequence to increase the frequency of unique byte values in the sub-sequence and adding to the index a data value representing the data transformation;
• on the sub-sequence having a freuency of unique byte values above a predefined threshold, adding to the index a data value representing the position of one or more unique byte values within the sub-sequence;
• creating an output data file, the output data file having a file type identifier;
• and
• adding the index to the output data file;
• and
• reversing the compression process to reconstruct the original data file.

A method for decompressing a data file, comprising:

• receiving a data file comprising a compressed original data file, the original data file comprising a sequence of bytes of a length greater than or equal to a predefined length and having been compressed by a compression process comprising:
• retrieving the original data file from a secondary storage device;
• storing the original data file in direct access memory;
• calculating the frequency of unique byte values within a sub-sequence of the original data file, the sub-sequence having a length not exceeding the predefined length;
• calculating the position of the one or more unique values within the sub-sequence;
• creating an index for the sub-sequence, the index including a data value representing the calculated frequency of unique byte values within the sub-sequence;
• and
• on the sub-sequence having a frequency of unique byte values below a predetermined threshold, applying a data transformation to the sub-sequence to increase the frequency of unique byte values in the sub-sequence and adding to the index a data value representing the data transformation;
• on the sub-sequence having a frequency of unique byte values above a predefined threshold, adding to the index a data value representing the position of one or more unique values within the sub-sequence;
• creating an output data file, the data file having a file type identifier;
• and
• adding the index to the output data file;
• and
• reversing the compression process to reconstruct the original data file.

A method for reconstructing an original data file, comprising:

• receiving a data file comprising a modified original data file having been modified by a modification process comprising:
• (a) reading the original data file into a sequence of multiple-byte data groups;
• (b) arranging the bytes of each data group in ascending byte value order;
• (c) writing an index into a first file that represents the original position of the bytes in each data group relative to their rearranged order;
• (d) writing the first byte value from each of the rearranged data groups into a second file;
• (e) writing calculated difference byte values representing the respective differences between each pair of consecutive byte values in each rearranged data group into a third file;
• and
• (f) outputting the first, second and third files for compression;
• and
• reversing the modification process to reconstruct the original data file.

A method for reconstructing an original data file, comprising:

• receiving a data file comprising a modified original data file, the original data file having a sequence of byte values and each byte value having an associated decimal order value representing its position in the sequence of the byte values of the original data file, the modified original data file having been modified by a modification process comprising:
• (a) writing the byte values having an even order value into a first file;
• (b) writing the byte values having an odd order value into a second file;
• and
• (c) outputting the first and second files for compression to a lossless compression algorithm for compression into a single file;
• and
• reversing the modification process to reconstruct the original data file.