# US 2015/0106549 A1 · Robust Data Replication

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0106549 A1
- Document ID: US20150106549A1
- Status: Published application
- Kind: A1
- Application number: 14/494,901
- Filed: 2014-09-24
- Published: 2015-04-16
- Earliest claimed: 2013-10-14
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Joanna K. Brown; Gordon J. Cockburn; Carlos F. Fuente; John P. Wilkinson
- Web page: https://patentgrep.com/patent/US20150106549A1

## CPC

- G06F 3/065: {Replication mechanisms}
- G06F 3/0619: {in relation to data integrity, e.g. data losses, bit errors}
- G06F 3/0679: {Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]}
- G06F 11/2064: {while ensuring consistency}
- G06F 11/2074: {Asynchronous techniques}
- G06F 11/2089: {Redundant storage control functionality}
- G06F 2206/1014: One time programmable [OTP] memory, e.g. PROM, WORM
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Disclosed is a system for replicating data. The system may comprise a plurality of nodes preferably organised in groups with one of the nodes acting as a coordinator node. The nodes are configured to receive write requests from an external server and to apply these write requests to a data storage source of the data storage system. The write requests typically belong to a batch of independent write actions identified by a batch sequence number. Each node stores the write request in non-volatile memory with the coordinator node monitoring which batches are secured in their entirety in non-volatile memory. The coordinator node authorises all other nodes to sequentially replicate the write requests in their non-volatile memory to the data storage source for all writes up to the highest batch sequence number for which all writes have been secured in non-volatile memory.

## Claims

### Claim 1

A data storage system (200) for replicating data from an external server (100) providing a sequence of batches comprising independent write actions, the data storage system comprising:
a data storage resource (230) comprising a plurality of data storage volumes; and
a plurality of nodes (220, 220′) communicatively coupled to each other and to the data storage resource, each node having access to a non-volatile memory and being adapted to receive a write request including a write action belonging to one of said batches and a batch sequence number identifying the position of said batch in said sequence from the external server and to store the write request into its non-volatile memory;
is the system comprising a further node adapted to act as a coordinator node (220′), said coordinator node being adapted to:
receive batch information from the external server indicating the batch sequence number and the number of write actions in said batch; and
periodically communicate with the other nodes to determine the highest batch sequence number for which all write actions have been written into the respective non-volatile memories and to communicate a first batch sequence number indicative of the highest batch sequence number to the other nodes;
wherein each node (220) is further adapted to:
store the communicated first batch sequence number;
replicate all write actions belonging to any batch having a sequence number not exceeding the communicated first batch sequence number from its non-volatile memory to a data storage volume in sequential order; and
upon completion of the replication, remove the corresponding write request from its non-volatile memory.

### Claim 2 (depends on 1)

The data storage system (200) of claim 1, wherein the nodes (220) are grouped into groups comprising an active node and at least one failover node, each group being responsible for replicating said write actions to a unique set of data storage volumes, wherein each node in the group is further adapted to:
store the write request into its non-volatile memory in response to a store instruction from one node in said group receiving the request; and
acknowledge completion of the write instruction to the one node; and
wherein one node in said group is further adapted to communicate to the coordinator node (220′) that the request is stored in the respective non-volatile memories of all nodes in said group upon receiving the acknowledgements from the other nodes in said group.

### Claim 3 (depends on 1)

The data storage system (200) of claim 1, wherein each node (220) is adapted to compare the respective target addresses of the write instructions stored in its non-volatile memory and to initiate replicating a next write instruction before the completion of the replication of a previous write instruction if the next write instruction has a unique target address in the non-volatile memory.

### Claim 4 (depends on 1)

The data storage system (200) of claim 1, wherein the first batch sequence number is the previous highest batch sequence number successfully transmitted to all other nodes (220); and wherein the coordinator node (220′) is further adapted to communicate the highest batch sequence number to all available nodes.

### Claim 5 (depends on 1)

The data storage system (200) of claim 1, wherein the coordinator node (220′) is further adapted to send a batch completion message for a particular batch to the external server (100) upon the highest batch sequence number exceeding the batch sequence number of the particular batch.

### Claim 6 (depends on 1)

The data storage system (200) of claim 1, wherein the further node is a node of said plurality of nodes, and wherein the system is adapted to dynamically assign the role of coordinator node (220′) to one of said nodes (220) at initialization or after failure of the previously assigned coordinator node.

### Claim 7 (depends on 1)

The data storage system (200) of claim 1, wherein the system is further adapted to select a highest batch sequence number stored in the non-volatile memory accessible by one of said nodes (220) upon the external server (100) resending one or more previously received write requests; and
wherein the coordinator node (220′) is adapted to communicate the selected highest batch sequence number to the other nodes (220); and
wherein each node is adapted to replicate all write actions belonging to any batch having a sequence number not exceeding the communicated selected first batch sequence number from its non-volatile memory to a data storage volume.

### Claim 8 (depends on 7)

The data storage system (200) of claim 7, wherein the system is further adapted to select the highest batch sequence number stored in the non-volatile memory accessible by said nodes (220).

### Claim 9 (depends on 1)

A data storage architecture including a primary data storage system (100) and a secondary data storage system comprising the data storage system (200) of any of claim 1 for backing up the primary data storage system, wherein the primary data storage system comprises:
a plurality of primary nodes (110) each adapted to issue a write request to one of the nodes (220) of the secondary data storage system; and
a primary server (110′) adapted to send batch information to the coordinator node (220′) of the secondary data storage system.

### Claim 10

A method of replicating data from an external server (100) providing a sequence of batches comprising independent write actions onto a data storage system (200) comprising a plurality of nodes (220, 220′) having access to a non-volatile memory and a data storage resource (230) comprising a plurality of data storage volumes, wherein each node is associated with a set of data volumes, the method comprising:
appointing one of said nodes (220) as a coordinator node (220′);
receiving, on the coordinator node, batch information from the external server indicating the batch sequence number and the number of write actions in said batch
receiving, on one of said nodes, a write request including a write action belonging to one of said batches and a batch sequence number identifying the position of said batch in said sequence from an external server and to store the write request into its non-volatile memory;
informing the coordinator node that the write request has been stored into the non-volatile memory;
determining, at the coordinator node, the highest batch sequence number for which all write actions have been written into the respective non-volatile memories;
communicating a first batch sequence number indicative of the highest batch sequence number from the coordinator node to the other nodes;
storing the communicated first batch sequence number in the non-volatile memories of all nodes receiving said communication; and for each node receiving said communication:
replicating all write actions belonging to any batch having a sequence number not exceeding the communicated first batch sequence number from its non-volatile memory to a data storage volume in sequential order; and
removing the corresponding write request from its non-volatile memory upon completion of the replication.

### Claim 11 (depends on 10)

The method of claim 10, wherein the nodes (220) are grouped into groups (210) comprising an active node and at least one failover node, each group being responsible for replicating said write actions to a unique set of data storage volumes, the method further comprising:
for each node in a group:
storing the write request into its non-volatile memory in response to a store instruction from one node in said group receiving the request; and
acknowledging completion of the write instruction to the one node; and
communicating to the coordinator node that the request is stored in the respective non-volatile memories of all nodes in said group upon receiving the acknowledgements from the other nodes in said group.

### Claim 12 (depends on 10)

The method of claim 10, further comprising, for each node (220):
comparing the respective target addresses of the write instructions stored in its non-volatile memory; and
initiating replication of a next write instruction before the completion of the replication of a previous write instruction if the next write instruction has a unique target address in the non-volatile memory.

### Claim 13 (depends on 10)

The method of claim 10, wherein the first batch sequence number is the previous highest batch sequence number successfully transmitted to all other nodes (220); the method further comprising communicating the highest batch sequence number from the coordinator node (220′) to all available nodes (220).

### Claim 14 (depends on 10)

The method of claim 10, further comprising sending a batch completion message for a particular batch from the coordinator node (220′) to the external server (100) upon the highest batch sequence number exceeding the batch sequence number of the particular batch.

### Claim 15 (depends on 10)

The method of claim 10, further comprising:
selecting a highest batch sequence number stored in the non-volatile memory of one of said nodes (220) upon the external server (100) resending one or more previously received write requests; and
communicating the selected highest batch sequence number from the coordinator node (220′) to the other nodes (220); and
at each node, replicating all write actions belonging to any batch having a sequence number not exceeding the communicated selected first batch sequence number from its non-volatile memory to a data storage volume.
