Crash recovery in a cluster network using distributed shared logging with multiple segment allocation

A method of maintaining data consistency in a cluster system using a distributed log, comprising:

• deploying a plurality of nodes in the cluster system, each node generating logs for system activity and maintained in a respective client, an associated client library, a respective log server, and a respective storage device comprising non-volatile RAM devices;
• storing the logs in a shared log structure that is segmented based on the logs and respective entries;
• maintaining in one node of the plurality of nodes, a metastore containing a mapping of the logs with respective storage locations in the respective storage devices;
• storing in a key value store (KVS) independent of the metastore, transactions captured by the logs as log transactions;
• replicating the log transactions as replicas;
• and
• replacing, upon an event impacting data integrity of any component in the system, writes after last consistent writes to the shared log structure with replicas.

A system storing logs in a cluster network to provide crash recovery and data consistency, comprising:

• a plurality of nodes deployed in the cluster network, each node comprising a respective client, an associated client library, a respective log server, and a respective storage device comprising non-volatile RAM devices;
• a shared log structure maintained in storage and storing the logs in a shared log structure that is segmented based on the logs and respective entries;
• a metastore component maintaining in one node of the plurality of nodes, a mapping of the logs with respective storage locations in the respective storage devices;
• and
• a detector detecting a crash of a component resulting in a crashed component in the cluster system;
• and
• a recovery component sealing logs of the crashed component, and activating a new instance of the crashed component, wherein the new instance continues writing logs in place of the crashed component.