Collecting and aggregating log data with fault tolerance

An apparatus for collecting and aggregating datasets for storage in a file system with fault tolerance, the apparatus including a memory storing instructions that, when executed by a processor of the apparatus, cause the apparatus to perform a method comprising:

• collecting datasets from a data source on a machine where the datasets are generated;
• generating, a batch comprising multiple messages from the datasets;
• assigning a tag to the batch and computing a checksum for the batch;
• writing the tag, the batch of multiple messages, and the checksum to an entry in a write-ahead-log (WAL) in storage;
• sending the datasets to a receiving location;
• in response to verifying the checksum of the batch of multiple messages at the receiving location, adding the tag to a map;
• writing a file to destination location;
• and
• identifying, in the file, tags associated with the batches in the file that have been written to the destination location.

A non-transitory computer-readable medium for collecting and aggregating datasets for storage in a file system with fault tolerance, the medium storing instructions that, when executed by a processor of an apparatus, cause the apparatus to:

• collecting datasets from a data source on a machine where the datasets are generated;
• generating, a batch comprising multiple messages from the datasets;
• assigning a tag to the batch and computing a checksum for the batch;
• writing the tag, the batch of multiple messages, and the checksum to an entry in a write-ahead-log (WAL) in storage;
• sending the datasets to a receiving location;
• in response to verifying the checksum of the batch of multiple messages at the receiving location, adding the tag to a map;
• writing a file to destination location;
• and
• identifying, in the file, tags associated with the batches in the file that have been written to the destination location.