Method and system for configuring a write amplification factor of a storage engine based on a compaction value associated with a data file

A non-transitory computer readable medium including computer executable instructions, wherein the instructions, when executed by a processor of a computer, cause the processor of the computer to perform a method for operating a storage engine, the method comprising:

• obtaining a compaction value associated with a data file, wherein the compaction value indicates the minimum number of records in the data file that are invalid to trigger a clean-up process of the data file, the data file is associated with an index file, which includes information with respect to each corresponding record included in the data file;
• retrieving a counter corresponding to the data file;
• and
• in response to the counter satisfying a first criterion,
• determining, for each record included in the data file, whether the record is to be written to another data file based on determining whether the record is a fresh record, wherein a write amplification factor of the storage engine is determined based on the compaction value.

A system having a processor, storage, and a communication platform capable of connecting to a network for operating a storage engine, the system comprising:

• a compaction unit implemented by the processor and configured to
• obtain a compaction value associated with a data file, wherein the compaction value indicates the minimum number of records in the data file that are invalid to trigger a clean-up process of the data file, the data file is associated with an index file, which includes information with respect to each corresponding record included in the data file;
• retrieve a counter corresponding to the data file;
• and
• in response to the counter satisfying a first criterion,
• determine, for each record included in the data file, whether the record is to be written to another data file based on determining whether the record is a fresh record, wherein a write amplification factor of the storage engine is determined based on the compaction value.