System, method, and computer program for maintaining consistency between a NoSQL database and non-transactional content associated with one or more files

A computer program product embodied on a non-transitory computer readable medium, comprising computer code for:

• identifying, by a system, one or more files including data resulting from events processed in an on-line-transactional-processing (OLTP) system, the events processed in the OLTP system further resulting in updates to the NoSQL database, and the one or more files being identified for maintaining consistency between the one or more files and the NoSQL database;
• storing, by the system, the data included in the one or more files as transactional data in a transaction file, wherein artifacts associated with the data are stored in a single data structure with an ending marker and the data is stored as the transactional data with a corresponding transaction ID;
• after storing the data in the transaction file, force flushing, by the system, the transaction file to guarantee the data is secured;
• storing, by the system, the transactional data of the transaction file in the NoSQL database associated with the system, wherein the transactional data of the transaction file is stored in the NoSQL database with the corresponding transaction ID;
• after storing the transactional data of the transaction file in the NoSQL database, identifying, by the system, each associated transaction as committed;
• scanning, by the system, the transaction file utilizing an asynchronous process to locate successful transactions from the transaction file, wherein a state of each transaction determined by querying the NoSQL database;
• and
• distributing, by the system, the artifacts associated with the successful transactions to their corresponding final locations;
• wherein the system logs success status and failure status of each transaction in the transaction file.

A system, comprising one or more processors, operable for:

• identifying, by the system, one or more files including data resulting from events processed in an on-line-transactional-processing (OLTP) system, the events processed in the OLTP system further resulting in updates to the NoSQL database, and the one or more files being identified for maintaining consistency between the one or more files and the NoSQL database;
• storing, by the system, the data included in the one or more files as transactional data in a transaction file, wherein artifacts associated with the data are stored in a single data structure with an ending marker and the data is stored as the transactional data with a corresponding transaction ID;
• after storing the data in the transaction file, force flushing, by the system, the transaction file to guarantee the data is secured;
• storing, by the system, the transactional data of the transaction file in the NoSQL database associated with the system, wherein the transactional data of the transaction file is stored in the NoSQL database with the corresponding transaction ID;
• after storing the transactional data of the transaction file in the NoSQL database, identifying, by the system, each associated transaction as committed;
• scanning, by the system, the transaction file utilizing an asynchronous process to locate successful transactions from the transaction file, wherein a state of each transaction determined by querying the NoSQL database;
• and
• distributing, by the system, the artifacts associated with the successful transactions to their corresponding final locations;
• wherein the system logs success status and failure status of each transaction in the transaction file.