Method and system for persisting data

A system for persisting and reconciling one or more data sources, the system comprising one or more processors and a data storage, the one or more processors configured to execute, or direct to be executed:

• an extract-transform-load module to:
• periodically extract batch data from at least one primary data source, the at least one primary data source not being real time in nature;
• transform the batch data into a first format specified by a data schema for a database;
• and
• load the transformed batch data into the database;
• a real-time distributed processing module to:
• continuously receive real-time data from at least one secondary data source;
• push the real-time data onto a distributed message queue system;
• index the received real-time data to generate an index;
• store the real-time data by key-value in a distributed key-value store for subsequent integration into the database;
• periodically extract the real-time data from the distributed key-value store and place the extracted real-time data into an interim data store;
• and
• transform the extracted real-time data stored in the interim data store into a second format specified by the data schema for the database;
• and
• at least one business intelligence tool to generate business intelligence reports comprising the real-time data in the index, the transformed real-time data in the database, the extracted batch data and the transformed batch data in the database.