Stream compression and decompression

An apparatus for compressing a sequence of records, each record comprising a sequence of fields, the method comprising:

• a memory;
• a processor communicatively coupled to the memory for performing:
• buffering a record in a line of a matrix;
• reordering the lines of the matrix according to locality sensitive hash values of the buffered records such that records with similar contents in corresponding fields are placed in proximity;
• consolidating fields in columns of the matrix into a block of codes;
• and
• consolidating yields codes of one of a first type comprising a sequence of individual fields and a second type comprising a sequence of fields with at least one repetition,
• wherein the second type of code comprises a presence field indicating repeated fields and an iteration field indicating the number of respective repetitions;
• the apparatus further comprising:
• decompressing records from a block of codes, for each code that is to be extracted from the block;
• determining the type of the code;
• decompressing the sequence of individual fields if the code is of the first type;
• and
• decompressing the sequence of repeated fields if the code is of the second type;
• a preceding determination of which codes must be decompressed for obtaining a predetermined record, wherein the fields are extracted into a matrix with lines representing records, the matrix is addressable in linearized form such that each field of each record has an offset address in the linearized matrix, and the preceding determination comprises the steps of:
• determining an offset address for each field of the predetermined record;
• linearly rescaling the determined offset addresses with a predetermined factor and setting a bit in a bitmap at the rescaled offset address to a predetermined value;
• for each code in the block:
• determining a range of offset addresses to which the code decompresses;
• linearly rescaling the range of offset addresses with the predetermined factor;
• and
• determining that the code must be decompressed if the bitmap contains a bit with the predetermined value at a rescaled address that falls within the rescaled range of offset addresses.