Consistent data delivery in an eventually consistent system

A processing system comprising one or more memories comprising computer-executable instructions;

• and one or more processors configured to execute the computer-executable instructions and cause the processing system to:
• store a first data table comprising a trigger timestamp;
• store a second data table comprising a commit timestamp and an adaptor offset corresponding to a latest change data capture (CDC) event recorded in a categorized data stream;
• determine that the commit timestamp is later than or the same as the trigger timestamp;
• store a third data table comprising a file writer offset corresponding to a latest data file of a plurality of data files stored in a file storage system;
• determine that the file writer offset is greater than or equal to the adaptor offset;
• cause the plurality of data files stored in the file storage system to be transmitted to and stored in a data lake as new files comprising transformed data extracted from the plurality of data files;
• and
• generate and transmit a notification comprising an indication that the data lake is up-to-date as of the trigger timestamp.

A computer-implemented method for delivering consistent data in an eventually consistent system, comprising:

• receiving a plurality of change data capture (CDC) events;
• storing the plurality of CDC events in a database;
• converting the plurality of CDC events to a plurality of data messages;
• generating a plurality of commit timestamps when the plurality of CDC events are converted to the plurality of data messages;
• publishing the plurality of data messages to a categorized data stream;
• generating a plurality of adaptor offsets corresponding to the plurality of data messages when the plurality of data messages is published to the categorized data stream;
• receiving a first notification that a latest commit timestamp of the plurality of commit timestamps is later than or the same as a trigger timestamp;
• transmitting the plurality of data messages to a file writer configured to read data messages and write corresponding files to a file storage system;
• writing a plurality of files corresponding to the plurality of data messages to the file storage system;
• generating a plurality of file writer offsets corresponding to the plurality of files written to the file storage system;
• receiving a second notification that a greatest file writer offset of the plurality of file writer offsets is greater than or equal to a greatest adaptor offset of the plurality of adaptor offsets;
• based on receiving the second notification that the greatest file writer offset of the plurality of file writer offsets is greater than or equal to the greatest adaptor offset of the plurality of adaptor offsets, transmitting the plurality of files to a data lake;
• and
• receiving a notification comprising an indication that the plurality of files have been transmitted to the data lake.