High-throughput data replication

A system comprising:

• memory;
• and
• at least one computing device configured with computer-executable instructions that, when executed, cause the at least one computing device to:
• receive a plurality of messages published by a first persistent message queue service, wherein the plurality of messages each relate to changes made to an on-premises database located remotely from a corresponding network-accessible database service, wherein the network-accessible database service stores at least a portion of data matching corresponding data that was stored in the on-premises database prior to the changes made to the on-premises database;
• encrypt portions of information within the plurality of messages at a field level;
• publish, via a second persistent message queue service, a plurality of encrypted messages that correspond to the plurality of messages, wherein the plurality of encrypted messages include portions of information encrypted at a field level;
• receive at least (a) a first batch of encrypted messages of the plurality of encrypted messages as published by the second persistent message queue service and (b) a second batch of encrypted messages of the plurality of encrypted messages as published by the second persistent message queue service;
• group the first batch of encrypted messages into a first plurality of sub-batches;
• group the second batch of encrypted messages into a second plurality of sub-batches;
• provide encrypted messages from at least the first plurality of sub-batches to a plurality of queues for parallel processing, wherein a different instance of a worker application or service is assigned to process each queue of encrypted messages;
• prior to processing completing for the first batch of encrypted messages, begin to provide encrypted messages from at least the second plurality of sub-batches to at least one of the plurality of queues for processing;
• subsequent to beginning to provide the encrypted messages from the at least the second plurality of sub-batches to the at least one of the plurality of queues for processing, determine that the first batch of encrypted messages have been fully processed;
• in response to the determining that the first batch of encrypted messages have been fully processed, commit changes related to the first batch of encrypted messages to corresponding data stored by the network-accessible database service;
• and
• in response to the determining that the second batch of encrypted messages have been fully processed, commit changes related to the second batch of encrypted messages to corresponding data stored by the network-accessible database service.

Non-transitory computer readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to at least:

• receive a plurality of messages published by a first persistent message queue service, wherein the plurality of messages each relate to changes made to an on-premises database located remotely from a corresponding network-accessible database service, wherein the network-accessible database service stores at least a portion of data matching corresponding data that was stored in the on-premises database prior to the changes made to the on-premises database;
• encrypt portions of information within the plurality of messages at a field level;
• publish, via a second persistent message queue service, a plurality of encrypted messages that correspond to the plurality of messages, wherein the plurality of encrypted messages include portions of information encrypted at a field level;
• receive at least (a) a first batch of encrypted messages of the plurality of encrypted messages as published by the second persistent message queue service and (b) a second batch of encrypted messages of the plurality of encrypted messages as published by the second persistent message queue service;
• group the first batch of encrypted messages into a first plurality of sub-batches;
• group the second batch of encrypted messages into a second plurality of sub-batches;
• provide encrypted messages from at least the first plurality of sub-batches to a plurality of queues for parallel processing, wherein a different instance of a worker application or service is assigned to process each queue of encrypted messages;
• prior to processing completing for the first batch of encrypted messages, begin to provide encrypted messages from at least the second plurality of sub-batches to at least one of the plurality of queues for processing;
• subsequent to beginning to provide the encrypted messages from the at least the second plurality of sub-batches to the at least one of the plurality of queues for processing, determine that the first batch of encrypted messages have been fully processed;
• in response to the determining that the first batch of encrypted messages have been fully processed, commit changes related to the first batch of encrypted messages to corresponding data stored by the network-accessible database service;
• and
• in response to the determining that the second batch of encrypted messages have been fully processed, commit changes related to the second batch of encrypted messages to corresponding data stored by the network-accessible database service.