Compression of tables based on occurrence of values

A method comprising:

• generating columns of dictionary-based compression values, the columns of dictionary-based compression values being based on a dictionary of possible values for each column of a column-based database;
• generating, by one or more processors, a bit vector for at least one of the columns, each of the bit vectors representing most-frequently occurring values of a respective column, the generating comprising having each bit of the bit vector represent whether the most-frequently occurring value exists in a respective row of the respective column;
• generating, by one or more processors, a number for each of the columns having an associated bit vector, the number representing an amount of occurrences of the most-frequently occurring value of one end of the column, wherein the most frequently occurring value comprises a plurality of bits;
• removing, by one or more processors, from each of the bit vectors a representation of the most-frequently occurring value of one end of the respective column based on the number associated with the bit vector;
• storing, by one or more processors, the number associated with each of the bit vectors to enable non-volatile memory searches of the data represented by the number associated with each of the bit vectors and each of the bit vectors;
• and
• generating a delta index separate from the columns that stores changes to at least one column occurring after the generation and storing of the compression values, wherein the stored changes comprises dictionary values being added to the delta index in chronological order to reflect an ordering of changes made to data in the columns over time.

A system comprising:

• at least one processor;
• and
• at least one memory including computer program code which when executed by the at least one processor causes operations comprising:
• generating columns of dictionary-based compression values, the columns of dictionary-based compression values being based on a dictionary of possible values for each column of a column-based database;
• generating a bit vector for at least one of the columns, each of the bit vectors representing most-frequently occurring values of a respective column, the generating comprising having each bit of the bit vector represent whether the most-frequently occurring value exists in a respective row of the respective column;
• generating a number for each of the columns having an associated bit vector, the number representing an amount of occurrences of the most-frequently occurring value of one end of the column, wherein the most frequently occurring value comprises a plurality of bits;
• removing from each of the bit vectors a representation of the most-frequently occurring value of one end of the respective column based on the number associated with the bit vector;
• storing the number associated with each of the bit vectors to enable non-volatile memory searches of the data represented by the number associated with each of the bit vectors and each of the bit vectors;
• and
• generating a delta index separate from the columns that stores changes to at least one column occurring after the generation and storing of the compression values, wherein the stored changes comprises dictionary being added to the delta index in chronological order to reflect an ordering of changes made to data in the columns over time.