Data storage using roaring binary-tree format

A method, comprising:

• maintaining, by a compute cluster having a set of compute nodes, a virtual data source (VDS) that is based on a origin data source, wherein the VDS includes a bitmap index that stores data ingested from the origin data source via an ingestion routine associated with the origin data source, and wherein the data of the bitmap index is stored in pages of a file that are mapped to nodes of a set of binary trees (b-trees);
• receiving, by the compute cluster, an update request to update the data of the bitmap index;
• and
• writing, by the compute cluster based on data that is identified by the update request, one or more new pages to a write-ahead log to replace corresponding pages of the file without having to rewrite the file;
• and
• processing a data request for data against the set of b-trees and the one or more new pages without accessing the origin data source.

A non-transitory computer-readable medium having program instructions stored thereon that are capable of causing a compute cluster having a set of compute nodes to perform operations comprising:

• maintaining a virtual data source (VDS) that is based on a origin data source, wherein the VDS includes a bitmap index that stores data ingested from the origin data source via an ingestion routine associated with the origin data source;
• receiving a data request for particular data stored in the bitmap index, wherein the data of the bitmap index is stored compressed in a plurality of containers included in pages that are nodes of a binary-tree (b-tree);
• accessing the requested particular data from the bitmap index without accessing the origin data source, wherein the accessing includes traversing the b-tree to extract one or more of the plurality of containers that correspond to the requested particular data;
• and
• returning a response to the data request that includes the requested particular data.