Compaction of documents in a high density data storage system

A non-transitory computer readable storage medium storing instructions that when executed by one or more computer processors, cause the one or more computer processors to perform steps for maintaining data in a data management system, the steps comprising:

• storing a plurality of documents in a log-structured object store, each document associated with a sequence number and a document value, wherein the log-structured object store comprises a plurality of log segments, each log segment storing a subset of documents that were written within a sequence number range;
• maintaining, in a first log-structured merge-tree, mapping document keys associated with documents to corresponding sequence numbers, the documents including active documents and stale documents;
• maintaining, in a second log-structured merge-tree, mapping sizes of stale documents to stale document sequence numbers;
• determining a ratio between a size of stale documents and a total size of all documents stored in at least one log segment exceeds a predetermined threshold based on the first log-structured merge-tree and the second log-structured merge-tree;
• performing a compaction operation on the log-structured object store, comprising:
• deleting stale documents in the log-structured object store;
• and
• consolidating active documents into fewer number of log segments using space freed up from the deleted stale documents;
• and
• updating the first log-structured merge-tree and the second log-structured merge-tree based on the compacted log segment.

A computer system comprising:

• one or more computer processors;
• and
• a non-transitory computer readable storage medium storing instructions that when executed by one or more computer processors, cause the one or more computer processors to perform steps for maintaining data in a data management system, the steps comprising:
• storing a plurality of documents in a log-structured object store, each document associated with a sequence number and a document value, wherein the log-structured object store comprises a plurality of log segments, each log segment storing a subset of documents that were written within a sequence number range;
• maintaining, in a first log-structured merge-tree, mapping document keys associated with documents to corresponding sequence numbers, the documents including active documents and stale documents;
• maintaining, in a second log-structured merge-tree, mapping sizes of stale documents to stale document sequence numbers;
• determining a ratio between a size of stale documents and a total size of all documents stored in at least one log segment exceeds a predetermined threshold based on the first log-structured merge-tree and the second log-structured merge-tree;
• performing a compaction operation on the log-structured object store, comprising:
• deleting stale documents in the log-structured object store;
• and
• consolidating active documents into fewer number of log segments using space freed up from the deleted stale documents;
• and
• updating the first log-structured merge-tree and the second log-structured merge-tree based on the compacted log segment.