Methods to maintain read-write consistency and dependent write order consistency within a cross-site storage system

A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a distributed storage system having a primary storage site and a secondary storage site, cause the one or more processing resources to:

• establish bi-directional synchronous replication between one or more members of a first storage node of the primary storage site and one or more members of a second storage node of the secondary storage site with each storage node having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO);
• receive, with the primary storage site, a write operation (Op) from a client device;
• write the write Op to a primary copy of data at the primary storage site and logging the write Op in an op log journal;
• terminate the write Op due to a network failure of a network, which causes a replication relationship to be out of sync, when attempting to replicate the write Op to the secondary storage site;
• determine a heartbeat failure when a time period for receiving a heartbeat message from the primary storage site expires;
• abort a replication engine and activating a fence on the storage node of the secondary storage site to disallow read/write access in response to the heartbeat failure;
• and
• upon network restoration, generate and send a consensus request from the primary storage site to a mediator at a third site to determine a primary role for serving input/output (I/O) operations for the primary storage site or the secondary storage site to avoid a split-brain scenario when the primary and secondary storage sites are temporarily unable to communicate with each other.

A distributed storage system comprising:

• one or more processing resource;
• and
• one or more non-transitory computer-readable media, coupled to the one or more processing resources, having stored therein instructions that when executed by the one or more processing resource cause the one or more processing resources to:
• establish bi-directional synchronous replication between one or more members of a first storage node of a primary storage site and one or more members of a second storage node of a secondary storage site with each storage node having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO) and with a primary storage site first replication principle;
• receive, with a first member of the first storage node of primary storage site, a first write operation (Op) from a client device;
• write the first write Op to a primary copy of data at the primary storage site and log the first write Op in an op log journal based on primary side first principle;
• terminate the first write Op due to a network failure of a network, which causes a synchronous replication relationship to be in out of sync (OOS) state, when attempting to replicate the first write Op to the secondary storage site;
• start a coordinated out of sync (OOS) procedure due to the first member of the primary storage site being out of sync (OOS) with the first member of the secondary storage site;
• receive, with a second member of the first storage node of the primary storage site, a second write Op that is dependent on the first write Op;
• write the second write Op to the primary copy of data at the primary storage site and log the second write Op in an op log journal;
• and
• preserve write order consistency by not replicating the second write Op to the secondary storage site based on the OOS state.