# US 12,008,018 B2 · Synchronous remote replication of snapshots

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,018 B2
- Document ID: US12008018B2
- Status: Granted
- Kind: B2
- Application number: 17/237,127
- Filed: 2021-04-22
- Published: 2022-10-27
- Granted: 2024-06-11
- Earliest claimed: 2021-04-22
- Assignees: EMC IP Holding Company LLC
- Applicants: EMC IP Holding Company LLC
- Inventors: Bhaskar Bora; Deepak Vokaliga; Mark J. Halstead; Jeffrey Wilson; William R. Stronge; Ian Adams
- Web page: https://patentgrep.com/patent/US12008018B2

## CPC

- G06F 16/1844: {Management specifically adapted to replicated file systems}
- G06F 16/275: {Synchronous replication}
- G06F 16/128: {Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion (error detection or correction of the data by redundancy in operations or in hardware G06F11/14, G06F11/16)}
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 11/1446: Point-in-time backing up or restoration of persistent data
- G06F 2201/84: Using snapshots, i.e. a logical point-in-time copy of the data
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Snapshots from a first LSU (R1) on a first storage system (A1) may be replicated to a second replica LSU (R2) on a second storage system (A2), for example, concurrently to remotely replicating (e.g., synchronously) write operations for R1 to R2. A process, P, on A1 executing the replication of the snapshots from R1 to R2 may be a separate process than the one or more processes on A1 executing remote replication of write operations for R1 to R2. During a consistency window on A1, outstanding write operations for R1 at the time the consistency window opened may be logged, and a pair of snapshots, SS11 and SS12 may be activated on R1 and R2, respectively. After the consistency window has closed, the SS12 snapshot metadata and snapshot data may be updated based on the outstanding write operations.

## Claims

### Claim 1

For a system including a host system, a first storage system, a second storage system, a first logical storage unit for which data is replicated from the first storage system to a second logical storage unit of the second storage system, a method of remotely replicating a first snapshot of the first logical storage unit from the first storage system to the second storage system, the method comprising:
while performing synchronous replication synchronously replicating write requests, that are directed to the first logical storage unit of the first storage system, to the second logical storage unit of the second data storage system, concurrently performing first processing that remotely replicates snapshots of the first logical storage unit to the second logical storage unit, said first processing including:
at a first point in time while performing said synchronous replication, the first storage system suspending initiating of processing of new write requests for the first logical storage unit that are received at the first storage system from the host system after the first point in time;
after said suspending, recording at least a first outstanding write request for the first logical storage unit on the first storage system, wherein each outstanding write request of the at least first outstanding write request is received from the host system at the first storage system and is outstanding at the first point in time when initiating processing of new write requests for the first logical storage unit is suspended by said suspending, and wherein said each outstanding write request is outstanding in that servicing of the said each outstanding write request has been initiated and the first storage system has not yet returned an acknowledgement to the host system regarding completion of said each outstanding write request;
after said recording, taking the first snapshot on the first storage system;
after said taking the first snapshot on the first storage system, the first storage system sending an instruction to the second storage system to initiate taking a second snapshot of the second logical storage unit;
responsive to the second storage system receiving, from the first storage system, the instruction to initiate the second storage system taking the second snapshot, the second storage system taking the second snapshot of the second logical storage unit;
responsive to taking the second snapshot, sending an acknowledgment from the second storage system to the first storage system regarding the taking of the second snapshot of the second logical storage unit; and
after the recording, the taking the first snapshot and the taking the second snapshot, performing processing including:
resuming the initiating of processing of new write requests for the first logical storage unit on the first storage system; and
applying the at least first outstanding write request to the second snapshot so that after performing said applying, the second snapshot is a remote replica of the first snapshot.

### Claim 2 (depends on 1)

The method of claim 1, wherein the applying is performed as part of a process executing independently of, and concurrently with, one or more processes that are remotely replicating write operations of the first logical storage unit to the second logical storage unit.

### Claim 3 (depends on 1)

The method of claim 1, wherein said applying the at least first outstanding write request includes, for each outstanding write request of the at least first outstanding write request:
determining whether the outstanding write request requires modification of snapshot data of the second snapshot; and
if it is determined that the outstanding write request requires modification of the second snapshot, modifying the snapshot data and snapshot metadata of the second snapshot on the second storage system based on the outstanding write request.

### Claim 4 (depends on 3)

The method of claim 3, wherein the first outstanding write request specifies a write operation for a first logical storage element corresponding to a second logical storage element of the second logical storage unit, and wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining whether first snapshot metadata of the first snapshot references a first current value of the first logical storage element on the first storage system;
determining whether second snapshot metadata of the second snapshot references a second current value of the second logical storage element on the second storage system; and
determining whether to modify the snapshot data and snapshot metadata of the second snapshot based at least in part on whether the first snapshot metadata references the first current value and whether the second snapshot metadata references the second current value.

### Claim 5 (depends on 4)

The method of claim 4, wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining that the first snapshot metadata references the first current value of the first logical storage element;
determining that the second snapshot metadata does not reference the second current value of the second logical storage element; and
modifying the snapshot metadata of the second snapshot to reference the second current value of the second logical storage element.

### Claim 6 (depends on 4)

The method of claim 4, wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining whether a first value of the second logical storage element specified by the second snapshot metadata is a same value as a second value of the second logical storage element specified by the write operation; and
determining whether to modify the snapshot data and snapshot metadata of the second snapshot based at least in part on whether the first value is the same value as the second value.

### Claim 7 (depends on 4)

The method of claim 4, wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining that the first snapshot metadata does not reference the first current value of the first logical storage element;
determining that the second snapshot metadata does not reference the second current value of the second logical storage element;
storing the second value of the second logical storage element at a location on the second storage system; and
modifying the snapshot metadata of the second snapshot to reference the location.

### Claim 8

A system comprising:
a first storage system;
a second storage system;
a third storage system;
a logical storage unit synchronously replicated between the first storage system and the second system, wherein the logical storage unit is asynchronously replicated from the first storage system to the third storage system and asynchronously replicated from the second storage system to the third storage system; and
executable logic that implements a method including:
while performing synchronous replication synchronously replicating write requests, that are directed to the first logical storage unit of the first storage system, to the second logical storage unit of the second data storage system, concurrently performing first processing that remotely replicates snapshots of the first logical storage unit to the second logical storage unit, said first processing including:
at a first point in time while performing said synchronous replication, the first storage system suspending initiating of processing of new write requests for the first logical storage unit that are received at the first storage system from the host system after the first point in time;
after said suspending, recording at least a first outstanding write request for the first logical storage unit on the first storage system, wherein each outstanding write request of the at least first outstanding write request is received from the host system at the first storage system and is outstanding at the first point in time when initiating processing of new write requests for the first logical storage unit is suspended by said suspending, and wherein said each outstanding write request is outstanding in that servicing of the said each outstanding write request has been initiated and the first storage system has not yet returned an acknowledgement to the host system regarding completion of said each outstanding write request;
after said recording, taking the first snapshot on the first storage system;
after said taking the first snapshot on the first storage system, the first storage system sending an instruction to the second storage system to initiate taking a second snapshot of the second logical storage unit;
responsive to the second storage system receiving, from the first storage system, the instruction to initiate the second storage system taking the second snapshot, the second storage system taking the second snapshot of the second logical storage unit;
responsive to taking the second snapshot, sending an acknowledgment from the second storage system to the first storage system regarding the taking of the second snapshot of the second logical storage unit; and
after the recording, the taking the first snapshot and the taking the second snapshot, performing processing including:
resuming the initiating of processing of new write requests for the first logical storage unit on the first storage system; and
applying the at least first outstanding write request to the second snapshot so that after performing said applying, the second snapshot is a remote replica of the first snapshot.

### Claim 9 (depends on 8)

The system of claim 8, wherein the applying is performed as part of a process executing independently of, and concurrently with, one or more processes that are remotely replicating write operations of the first logical storage unit to the second logical storage unit.

### Claim 10 (depends on 8)

The system of claim 8, wherein said applying the at least first outstanding write request includes, for each outstanding write request of the at least first outstanding write request:
determining whether the outstanding write request requires modification of snapshot data of the second snapshot; and
if it is determined that the outstanding write request requires modification of the second snapshot, modifying the snapshot data and snapshot metadata of the second snapshot on the second storage system based on the outstanding write request.

### Claim 11 (depends on 8)

The system of claim 8, wherein the first outstanding write request specifies a write operation for a first logical storage element corresponding to a second logical storage element of the second logical storage unit, and wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining whether first snapshot metadata of the first snapshot references a first current value of the first logical storage element on the first storage system;
determining whether second snapshot metadata of the second snapshot references a second current value of the second logical storage element on the second storage system; and
determining whether to modify the snapshot data and snapshot metadata of the second snapshot based at least in part on whether the first snapshot metadata references the first current value and whether the second snapshot metadata references the second current value.

### Claim 12 (depends on 11)

The system of claim 11, wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining that the first snapshot metadata references the first current value of the first logical storage element;
determining that the second snapshot metadata does not reference the second current value of the second logical storage element; and
modifying the snapshot metadata of the second snapshot to reference the second current value of the second logical storage element.

### Claim 13 (depends on 11)

The system of claim 11, wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining whether a first value of the second logical storage element specified by the second snapshot metadata is a same value as a second value of the second logical storage element specified by the write operation; and
determining whether to modify the snapshot data and snapshot metadata of the second snapshot based at least in part on whether the first value is the same value as the second value.

### Claim 14 (depends on 11)

The system of claim 11, wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining that the first snapshot metadata does not reference the first current value of the first logical storage element;
determining that the second snapshot metadata does not reference the second current value of the second logical storage element;
storing the second value of the second logical storage element at a location on the second storage system; and
modifying the snapshot metadata of the second snapshot to reference the location.

### Claim 15

For a system including a host system, a first storage system, a second storage system, a first logical storage unit for which data is replicated from the first storage system to a second logical storage unit of the second storage system, one or more non-transitory computer-readable media having code stored thereon that, when executed, performs a method of remotely replicating a first snapshot of the first logical storage unit from the first storage system to the second storage system comprising:
while performing synchronous replication synchronously replicating write requests, that are directed to the first logical storage unit of the first storage system, to the second logical storage unit of the second data storage system, concurrently performing first processing that remotely replicates snapshots of the first logical storage unit to the second logical storage unit, said first processing including:
at a first point in time while performing said synchronous replication, the first storage system suspending initiating of processing of new write requests for the first logical storage unit that are received at the first storage system from the host system after the first point in time;
after said suspending, recording at least a first outstanding write request for the first logical storage unit on the first storage system, wherein each outstanding write request of the at least first outstanding write request is received from the host system at the first storage system and is outstanding at the first point in time when initiating processing of new write requests for the first logical storage unit is suspended by said suspending, and wherein said each outstanding write request is outstanding in that servicing of the said each outstanding write request has been initiated and the first storage system has not yet returned an acknowledgement to the host system regarding completion of said each outstanding write request;
after said recording, taking the first snapshot on the first storage system;
after said taking the first snapshot on the first storage system, the first storage system sending an instruction to the second storage system to initiate taking a second snapshot of the second logical storage unit;
responsive to the second storage system receiving, from the first storage system, the instruction to initiate the second storage system taking the second snapshot, the second storage system taking the second snapshot of the second logical storage unit;
responsive to taking the second snapshot, sending an acknowledgment from the second storage system to the first storage system regarding the taking of the second snapshot of the second logical storage unit; and
after the recording, the taking the first snapshot and the taking the second snapshot, performing processing including:
resuming the initiating of processing of new write requests for the first logical storage unit on the first storage system; and
applying the at least first outstanding write request to the second snapshot so that after performing said applying, the second snapshot is a remote replica of the first snapshot.

### Claim 16 (depends on 15)

The one or more non-transitory computer-readable media of claim 15, wherein said applying the at least first outstanding write request includes, for each outstanding write request of the at least first outstanding write request:
determining whether the outstanding write request requires modification of snapshot data of the second snapshot; and
if it is determined that the outstanding write request requires modification of the second snapshot, modifying the snapshot data and snapshot metadata of the second snapshot on the second storage system based on the outstanding write request.

### Claim 17 (depends on 16)

The one or more non-transitory computer-readable media of claim 16, wherein the first outstanding write request specifies a write operation for a first logical storage element corresponding to a second logical storage element of the second logical storage unit, and wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining whether first snapshot metadata of the first snapshot references a first current value of the first logical storage element on the first storage system;
determining whether second snapshot metadata of the second snapshot references a second current value of the second logical storage element on the second storage system; and
determining whether to modify the snapshot data and snapshot metadata of the second snapshot based at least in part on whether the first snapshot metadata references the first current value and whether the second snapshot metadata references the second current value.

### Claim 18 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining that the first snapshot metadata references the first current value of the first logical storage element;
determining that the second snapshot metadata does not reference the second current value of the second logical storage element; and
modifying the snapshot metadata of the second snapshot to reference the second current value of the second logical storage element.

### Claim 19 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining whether a first value of the second logical storage element specified by the second snapshot metadata is a same value as a second value of the second logical storage element specified by the write operation; and
determining whether to modify the snapshot data and snapshot metadata of the second snapshot based at least in part on whether the first value is the same value as the second value.

### Claim 20 (depends on 17)

The one or more non-transitory computer-readable media of claim 17, wherein said determining whether the first outstanding write request requires modification of snapshot data of the second snapshot includes:
determining that the first snapshot metadata does not reference the first current value of the first logical storage element;
determining that the second snapshot metadata does not reference the second current value of the second logical storage element;
storing the second value of the second logical storage element at a location on the second storage system; and
modifying the snapshot metadata of the second snapshot to reference the location.