Methods, apparatus and computer programs for data replication

An apparatus for data replication, comprising:

• two or more data processing systems, each having a processing unit, a storage unit with data stored therein and a message transmission unit;
• at least one of said data processing systems having a capture program for capturing transaction data changes and batching transaction change descriptions;
• and at least one of said data processing systems having a plurality of change agents for applying data change descriptions to data in the storage unit and an apply browser for reading data change descriptions, resolving conflicts and assigning data change descriptions to the apply agents;
• whereby the apparatus:
• capturing a description of changes to a first data instance, which changes are performed as source transactions at a first data processing system;
• combining the captured descriptions of change for a plurality of source transactions, to form a batch of source transactions;
• transmitting the batch of source transactions to a second data processing system wherein the batch is transmitted to the second data processing system within a message;
• receiving the transmitted batch at the second data processing system and separating the descriptions of changes from the batch into sets of changes corresponding to their respective source transactions, wherein the message is held in persistent storage at the second data processing system until a check of transaction processed records determines that apply transactions have been completed for all of the plurality of source transactions;
• and
• for each separate source transaction, applying the corresponding set of changes to a data replica at the second data processing system as a respective separate apply transaction and writing a transaction processed record to indicate completion of the respective apply transaction;
• and further comprising:
• a message cleaner periodically checking the transaction processed records to identify messages for which apply transactions have been completed for all of the respective plurality of source transactions;
• the message cleaner checking whether deletion of any of the said identified messages would result in holes in the sequence of messages held in persistent storage;
• and
• the message cleaner then deleting only a subset of said identified messages so as to avoid holes in said sequence of messages.

A computer program product comprising program code recorded on a non-transitory, computer-readable recording medium for controlling applying data changes to a data replica, comprising:

• program code for capturing a description of changes to a first data instance, which changes are performed as source transactions at a first data processing system;
• program code for combining the captured descriptions of change for a plurality of source transactions, to form a batch of source transactions;
• program code for transmitting the batch of source transactions to a second data processing system, wherein the batch is transmitted to the second data processing system within a message;
• program code for receiving the transmitted batch at the second data processing system and separating the descriptions of changes from the batch into sets of changes corresponding to their respective source transactions, wherein the message is held in persistent storage at the second data processing system until a check of transaction processed records determines that apply transactions have been completed for all of the plurality of source transactions;
• and
• program code for, for each separate source transaction, applying the corresponding set of changes to a data replica at the second data processing system as a respective separate apply transaction and writing a transaction processed record to indicate completion of the respective apply transaction;
• program code for periodically checking the transaction processed records to identify messages for which apply transactions have been completed for all of the respective plurality of source transactions;
• program code for checking whether deletion of any of the said identified messages would result in holes in the sequence of messages held in persistent storage;
• and
• program code for deleting only a subset of said identified messages so as to avoid holes in said sequence of messages.