Verifying Consistency Of Replicated Datasets

One or more non-transitory computer-readable media comprising instructions that, when executed by one or more hardware processors, cause performance of operations comprising:

• executing a replication process comprising replicating source data from a source dataset stored in a source data repository to a destination data repository comprising a destination dataset, wherein replicating the source data comprises transmitting data messages, comprising the source data, over a replication stream between a source entity associated with the source data repository and a destination entity associated with the destination data repository;
• executing a first sampling process comprising obtaining a first source data sample from the source dataset, the first source data sample corresponding to source data previously transmitted to the destination entity for replication of at least a portion of the source dataset;
• generating a first control message comprising the first source data sample obtained via the first sampling process;
• inserting the first control message between two or more data messages being transmitted over the replication stream;
• transmitting the first control message to the destination entity over the replication stream contemporaneously with the two or more data messages, wherein the first control message signals a consistency verification process to be executed by the destination entity during the replication process;
• wherein the destination entity receives the first control message over the replication stream contemporaneously with the two or more data messages and responsive to receiving the first control message, the destination entity extracts the first source data sample from the first control message and executes a first verification process comprising comparing the first source data sample to at least a first portion of the destination dataset to verify consistency between the first source data sample and at least the first portion of the destination dataset;
• wherein the destination entity classifies the source dataset and the destination dataset as consistent based at least in part on verifying consistency between the first source data sample and at least the first portion of the destination dataset.

A system comprising:

• at least one device including a hardware processor;
• the system being configured to perform operations comprising:
• executing a replication process comprising replicating source data from a source dataset stored in a source data repository to a destination data repository comprising a destination dataset, wherein replicating the source data comprises transmitting data messages, comprising the source data, over a replication stream between a source entity associated with the source data repository and a destination entity associated with the destination data repository;
• executing a first sampling process comprising obtaining a first source data sample from the source dataset, the first source data sample corresponding to source data previously transmitted to the destination entity for replication of at least a portion of the source dataset;
• generating a first control message comprising the first source data sample obtained via the first sampling process;
• inserting the first control message between two or more data messages being transmitted over the replication stream;
• transmitting the first control message to the destination entity over the replication stream contemporaneously with the two or more data messages, wherein the first control message signals a consistency verification process to be executed by the destination entity during the replication process;
• wherein the destination entity receives the first control message over the replication stream contemporaneously with the two or more data messages and responsive to receiving the first control message, the destination entity extracts the first source data sample from the first control message and executes a first verification process comprising comparing the first source data sample to at least a first portion of the destination dataset to verify consistency between the first source data sample and at least the first portion of the destination dataset;
• wherein the destination entity classifies the source dataset and the destination dataset as consistent based at least in part on verifying consistency between the first source data sample and at least the first portion of the destination dataset.