Look-Ahead Staging for Accelerated Data Extraction

A system, comprising:

• a memory;
• and
• at least one processor coupled to the memory and configured to:
• receive a data change for a data extraction from a producer job at a source system;
• store the data change in a staging area of a persistent storage together with a respective sequence identifier, wherein the staging area corresponds to the data extraction;
• receive a request for a next package of data changes in the staging area from a consumer job at a target system;
• generate a next package identifier based on a next package identifier in a control area of the persistent storage, wherein the generating the next package identifier is performed atomically;
• mark a set of data changes in the staging area with the next package identifier based on a maximum package size in the control area, wherein the marking the set of data changes is performed atomically;
• generate the next package based on the set of data changes marked with the next package identifier;
• transmit the next package to the consumer job;
• receive a commit notification for the next package from the consumer job in response to the transmitting;
• and
• remove the data changes in the next package from the staging area in response to receiving the commit notification for the next package.

A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:

• receiving a data change for a data extraction from a producer job at a source system;
• storing the data change in a staging area of a persistent storage together with a respective sequence identifier, wherein the staging area corresponds to the data extraction;
• receiving a request for a next package of data changes in the staging area from a consumer job at a target system;
• generating a next package identifier based on a next package identifier in a control area of the persistent storage, wherein the generating the next package identifier is performed atomically;
• marking a set of data changes in the staging area with the next package identifier based on a maximum package size in the control area, wherein the marking the set of data changes is performed atomically;
• generating the next package based on the set of data changes marked with the next package identifier;
• transmitting the next package to the consumer job;
• receiving a commit notification for the next package from the consumer job in response to the transmitting;
• and
• removing the data changes in the next package from the staging area in response to receiving the commit notification for the next package.