Data compression in a multi-layer distributed datastore

A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to perform a method for compressing data received to be stored in a set of one or more physical disks of a set of one or more host machines, the method comprising:

• receiving a data block of data to store on at least one physical disk in the set of the one or more physical disks of the set of the one or more host machines;
• determining whether to store the data of the data block in a data log on the at least one physical disk as uncompressed data or compressed data based on whether a determined size of resulting compressed data when compressing the data would allow for metadata about the resulting compressed data to be stored with the resulting compressed data in the data log;
• when determining to store the data as the compressed data:
• compressing the data of the data block to generate the compressed data;
• and
• storing the compressed data in at least one sector in the data log;
• when determining to store the data as the uncompressed data:
• storing the data, uncompressed, in a plurality of sectors in the data log;
• generating a plurality of state bits, the plurality of state bits indicating whether the data of the data block is stored in the data log as the uncompressed data or the compressed data, and when the data of the data block is stored in the data log as the compressed data, the plurality of state bits indicating a size of the compressed data;
• and
• storing the plurality of state bits in an entry of a logical map table, the entry associated with a logical block address (LBA) that corresponds to the data block.

A computer system, comprising:

• a memory;
• and
• a processor coupled to the memory, the processor being configured to:
• receive a data block of data to store on at least one physical disk in a set of physical disks of a set of host machines;
• determine whether to store the data of the data block in a data log on the at least one physical disk as uncompressed data or compressed data based on whether a determined size of resulting compressed data when compressing the data would allow for metadata about the resulting compressed data to be stored with the resulting compressed data in the data log;
• when determining to store the data as the compressed data:
• compress the data of the data block to generate the compressed data;
• and
• store the compressed data in at least one sector in the data log;
• when determining to store the data as the uncompressed data:
• store the data, uncompressed, in a plurality of sectors in the data log;
• generate a plurality of state bits, the plurality of state bits indicating whether the data of the data block is stored in the data log as the uncompressed data or the compressed data, and when the data of the data block is stored in the data log as the compressed data, the plurality of state bits indicating a size of the compressed data;
• and
• store the plurality of state bits in an entry of a logical map table, the entry associated with a logical block address (LBA) that corresponds to the data block.