# US 2025/0165177 A1 · Usage of op log based fast recovery of bidirectional synchronous replication in a cross-site distributed storage system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0165177 A1
- Document ID: US20250165177A1
- Status: Published application
- Kind: A1
- Application number: 19/034,412
- Filed: 2025-01-22
- Published: 2025-05-22
- Earliest claimed: 2021-05-05
- Assignees: NetApp, Inc.
- Applicants: NetApp, Inc.
- Inventors: Vishal Thakur; Akhil Kaushik; Anoop Vijayan; Krishna Murthy Chandraiah Setty Narasingarayanapeta
- Web page: https://patentgrep.com/patent/US20250165177A1

## CPC

- G06F 3/065: {Replication mechanisms}
- G06F 3/0613: {in relation to throughput}
- G06F 3/0619: {in relation to data integrity, e.g. data losses, bit errors}
- G06F 3/0631: {by allocating resources to storage systems}
- G06F 3/0653: {Monitoring storage devices or systems}
- G06F 3/0659: {Command handling arrangements, e.g. command buffers, queues, command scheduling}
- G06F 3/067: {Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

In one embodiment, a computer-implemented method comprises establishing bi-directional synchronous replication between one or more storage objects of a primary storage site and one or more storage objects of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO), initiating a resynchronization process due to a loss of the bi-directional synchronous replication between the one or more storage objects and the one or more storage objects, and performing the resynchronization process based on using inflight tracking replay and reconciliation between a first Op log metafile of the primary storage site and a second Op log metafile of the secondary storage site.

## Claims

### Claim 1

A computer-implemented method performed by one or more processing resources of a distributed storage system, the computer-implemented method comprising:
establishing bi-directional synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO);
initiating a resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 due to a loss of the bi-directional synchronous replication between the one or more storage objects of the CG1 and the one or more storage objects of the CG2; and
performing the resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 based on using inflight tracking replay and reconciliation between a first Op log metafile of the primary storage site and a second Op log metafile of the secondary storage site.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprises:
designating the primary storage site as a global sequence number generator to ensure unique sequence numbers for operations received on each of the primary and secondary storage sites; and
maintaining counters for Operation Sequence Number and Cumulative Response Sequence Number (CSN) on the primary storage site.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, further comprises:
maintaining a counter for CSN on the secondary storage site.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprises:
upon operations for one or more storage objects of CG1 being received on the primary storage site, assigning the received operations an Op Sequence Number and CSN;
sending the operations having the Op Sequence Number and CSN from the primary storage site to the secondary storage site; and
receiving, with the primary storage site, a response from the secondary storage site having a CSN for the secondary storage site.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, further comprises:
comparing the CSN of the primary storage site to the received CSN from the secondary storage site; and
updating the CSN to be equal to the received CSN when the CSN of the primary storage site is less than the received CSN from the secondary storage site.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, further comprises:
receiving operations on the secondary storage site for one or more storage objects of CG2;
sending the received operations with a CSN of the secondary storage site to the primary storage site;
sending a response from the primary storage site having an Op Sequence Number that is generated on the primary storage site and a CSN of the primary storage site to the secondary storage site; and
updating the CSN of the secondary storage site to be equal to the CSN of the primary storage site when the CSN of secondary storage site is lower than the received CSN.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprises:
initiating the inflight tracking and reconciliation of I/O operations based on the first Op log metafile of the primary storage site and the second Op log metafile of the secondary storage site.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, further comprises:
replaying the I/O operations from one of the first and second Op log metafiles; and
reconciling the I/O operations in the first Op log metafile with the I/O operations in the second Op log metafile by determining whether an Op being considered is missing from one of the first and second Op log metafiles and sending the missing Op from the primary storage site to the secondary storage site when the Op is missing from the second Op log metafile.

### Claim 9

A non-transitory computer-readable storage medium embodying a set of instructions, which when executed by one or more processing resources of a multi-site distributed storage system cause the one or more processing resources to:
establish bi-directional synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO);
initiate a resynchronization process between the one or more storage objects of the CG1and the one or more storage objects of the CG2 due to a loss of the bi-directional synchronous replication between the one or more storage objects of the CG1 and the one or more storage objects of the CG2; and
perform the resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 based on using inflight tracking replay and reconciliation between a first Op log metafile of the primary storage site and a second Op log metafile of the secondary storage site.

### Claim 10 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
designate the primary storage site as a global sequence number generator to ensure unique sequence numbers for operations received on each of the primary and secondary storage sites;
maintain counters for Operation Sequence Number and Cumulative Response Sequence Number (CSN) on the primary storage site; and
maintain a counter for CSN on the secondary storage site.

### Claim 11 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
upon operations for one or more storage objects of CG1 being received on the primary storage site, assign the received operations an Op Sequence Number and CSN;
send the operations having the Op Sequence Number and CSN from the primary storage site to the secondary storage site; and
receive, with the primary storage site, a response from the secondary storage site having a CSN for the secondary storage site.

### Claim 12 (depends on 11)

The non-transitory computer-readable storage medium of claim 11, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
compare the CSN of the primary storage site to the received CSN from the secondary storage site; and
update the CSN to be equal to the received CSN when the CSN of the primary storage site being less than the received CSN from the secondary storage site.

### Claim 13 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
receive operations on the secondary storage site for one or more storage objects of CG2;
send the received operations with a CSN of the secondary storage site to the primary storage site;
send a response from the primary storage site having an Op Sequence Number that is generated on the primary storage site and a CSN of the primary storage site to the secondary storage site; and
update the CSN of the secondary storage site to be equal to the CSN of the primary storage site when the CSN of secondary storage site CSN is lower than the received CSN.

### Claim 14 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
initiate the inflight tracking and reconciliation of I/O operations based on the first Op log metafile of the primary storage site and the second Op log metafile of the secondary storage site;
replay the I/O operations from one of the first and second Op log metafiles; and
reconcile the I/O operations in the first Op log metafile with the I/O operations in the second Op log metafile by determining whether an Op being considered is missing from one of the first and second Op logs and sending the missing Op from the primary storage site to the secondary storage site when the Op is missing from the second Op log metafile.

### Claim 15

A multi-site storage system having a primary storage site with a primary storage cluster and a secondary storage site with a secondary storage cluster, comprising:
one or more processing resources; and
a non-transitory computer-readable medium coupled to the one or more processing resources, having stored therein instructions, which when executed by the one or more processing resources cause the one or more processing resources to:
establish bi-directional synchronous replication between one or more storage objects of a first consistency group (CG1) of a primary storage site and one or more storage objects of a second consistency group (CG2) of a secondary storage site with each storage site having read/write access while maintaining zero recovery point objective (RPO) and Zero recovery time objective (RTO);
initiate a resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 due to a loss of the bi-directional synchronous replication between the one or more storage objects of the CG1 and the one or more storage objects of the CG2; and
perform the resynchronization process between the one or more storage objects of the CG1 and the one or more storage objects of the CG2 based on using inflight tracking replay and reconciliation between a first Op log metafile of the primary storage site and a second Op log metafile of the secondary storage site.

### Claim 16 (depends on 15)

The multi-site storage system of claim 15, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
designate the primary storage site as a global sequence number generator to ensure unique sequence numbers for operations received on each of the primary and secondary storage sites;
maintain counters for Operation Sequence Number and Cumulative Response Sequence Number (CSN) on the primary storage site; and
maintain a counter for CSN on the secondary storage site.

### Claim 17 (depends on 15)

The multi-site storage system of claim 15, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
upon operations for one or more storage objects of CG1 being received on the primary storage site, assign the received operations an Op Sequence Number and CSN;
send the operations having the Op Sequence Number and CSN from the primary storage site to the secondary storage site; and
receive, with the primary storage site, a response from the secondary storage site having a CSN for the secondary storage site.

### Claim 18 (depends on 15)

The multi-site storage system of claim 15, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
receive operations on the secondary storage site for one or more storage objects of CG2;
send the received operations with a CSN of the secondary storage site to the primary storage site;
send a response from the primary storage site having an Op Sequence Number that is generated on the primary storage site and a CSN of the primary storage site to the secondary storage site; and
update the CSN of the secondary storage site to be equal to the CSN of the primary storage site when the CSN of secondary storage site CSN is lower than the received CSN.

### Claim 19 (depends on 15)

The multi-site storage system of claim 15, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
initiate the inflight tracking and reconciliation of I/O operations based on the first Op log metafile of the primary storage site and the second Op log metafile of the secondary storage site.

### Claim 20 (depends on 15)

The multi-site storage system of claim 15, wherein the instructions when executed by the one or more processing resources cause the one or more processing resources to:
replay I/O operations from one of the first and second Op log metafiles; and
reconcile the I/O operations in the first Op log metafile with the I/O operations in the second Op log metafile by determining whether an Op being considered is missing from one of the first and second Op logs and sending the missing Op from the primary storage site to the secondary storage site when the Op is missing from the second Op log metafile.
