Techniques and Architectures for Providing an Extract-Once Framework Across Multiple Data Sources

The method of claim 14 wherein the data collection platform comprises a data lake.

A system comprising:

• a data storage device;
• one or more processors coupled with the data storage device, the one or more processors to start a data consumption job to ingest data from at least one of the disparate heterogenous sources to multiple tables within the data collection platform, the data consumption job having multiple processes for processing and writing data to at least one of the multiple tables, to retrieve checkpoint metadata corresponding to the data consumption job from a checkpoint metadata store, to perform a subset of processes from the data consumption job, to update the checkpoint metadata in the checkpoint metadata store corresponding to the data consumption job in response to completion of the subset of processes from the data consumption job, to perform at least one subsequent subset of processes from the data consumption job, to update the checkpoint metadata in the checkpoint metadata store corresponding to the data consumption job in response to completion of each of the at least one subsequent subset of processes from the data consumption job, and to update batch metadata in a batch metadata store in response to completion of the data consumption job.