Fully pipelined hardware engine design for fast and efficient inline lossless data compression

A hardware pipeline data compression system, comprising:

• a data buffer;
• a first hash table, implemented across a first plurality of blocks of RAM (random access memory) arranged for parallel access of a plurality of hash table entries;
• a second hash table, implemented across a second plurality of blocks of RAM arranged for parallel access of a further plurality of hash table entries;
• a string match module comprising a plurality of comparators and counters each to compare and count a number of consecutive byte matches of pre-compression data of one location and pre-compression data of another location, for a plurality of comparisons in parallel;
• a hash lookup module, implemented in hardware logic, to perform multiple hash operations in parallel on the pre-compression data, perform the parallel access of the plurality of hash table entries in the first hash table, and perform the parallel access of the further plurality of hash table entries in the second hash table;
• and
• a hardware pipeline arranged to receive the pre-compression data into the data buffer, populate the first hash table and the second hash table, supply the hash lookup module and the string match module, and generate compression data based on results from the string match module.

A method of data compression, practiced by a hardware compression system, the method comprising:

• writing pre-compression data into a data buffer;
• populating a first hash table and a second hash table;
• supplying the pre-compression data to a hash lookup module to form hash results and make a plurality of accesses to the first hash table and a plurality of accesses to the second hash table, in parallel;
• supplying a string match module with results from the hash lookup module so that the string match module compares pre-compression data from multiple locations in the data buffer in parallel;
• generating, in a match merge module, literals and metadata for compression data based on results from the string match module;
• and
• encoding the compression data according to a data compression format, based on the literals and the metadata from the match merge module.