Compression algorithm incorporating automatic generation of a bank of predefined huffman dictionaries

A hybrid static/dynamic method of generating a bank of Huffman dictionaries for a group of files for use in an encoder, said method comprising, said method comprising:

• randomly selecting a subset of files from said group of files;
• for each file in said subset of files selected from said group of files, applying a repetition removal algorithm thereto;
• collecting literal frequencies per file and computing the correlation between input data blocks to verify that they do not significantly differ;
• amortizing the frequencies from all blocks and translating frequencies resulting from said repetition removal algorithm to probabilities and amortizing said probabilities over said subset of files;
• generating a unified probability list, translating said unified probability list back to frequencies and turning zero frequencies into low non-zero values;
• and
• building a dynamic Huffman tree and table based on said list of frequencies.

A hybrid static/dynamic method of generating a bank of Huffman dictionaries for a group of files, said method comprising for use in an encoder, said method comprising:

• randomly selecting a subset of files from said group of files;
• for each file in said subset of files selected from said group of files, applying a repetition removal algorithm thereto;
• gathering literal frequencies for each file in said subset and computing the correlation between input data blocks to verify that they do not significantly differ, and if they do significantly differ, replacing the corresponding file with replacement file from said group of files;
• amortizing the frequencies from all blocks and translating frequencies resulting from said repetition removal algorithm to probabilities and amortize said probabilities over said subset of files;
• if a majority of non-negligible probabilities are significantly similar, verifying original probabilities in each file and enforcing the same probability for all similar values if the distribution of those literals is substantially uniform;
• translating said unified probability list back to frequencies;
• converting zero value frequencies into low non-zero values;
• and
• building a dynamic Huffman tree and table based on said list of frequencies.