A data pipeline system for ingesting data into cloud storage, the system comprising:
- one or more processors;
- and
- computer-readable memory encoded with instructions that, when executed by the one or more processors, cause the data pipeline system to:
- consolidate pre-entered metadata enabling an identification of multiple data sources of multiple data pipelines, and a connection to each of the multiple data sources, into a common metadata table for an overarching data pipeline prior to initiating the overarching data pipeline, wherein the pre-entered metadata of the metadata table includes values associated with each of the multiple data sources that specify, for each of the multiple data sources, at least one of:
- a data source identification;
- a data source name;
- whether a respective one of the multiple data sources is enabled;
- and
- a key value that is associated with access credentials for the respective one of the multiple data sources;
- initiate the overarching data pipeline by performing a lookup activity, the lookup activity including querying the metadata table that includes the pre-entered metadata, wherein the querying specifies a plurality of data sources of the multiple data sources;
- create an overarching pipeline start log corresponding to a start of the overarching data pipeline when the overarching data pipeline is initiated;
- retrieve, based on the querying, metadata that corresponds to the specified plurality of data sources of the multiple data sources;
- and
- perform a sequence of activities for each data source of the specified data sources, the sequence of activities including:
- creating a data source pipeline start log corresponding to a start of the sequence of activities for a respective one of the specified plurality of data sources before forming a connection to the respective one of the specified plurality of data sources;
- connecting to the respective one of the specified plurality of data sources using a portion of the metadata from the metadata table that corresponds to the respective one of the specified plurality of data sources to form the connection to the respective one of the specified data sources by:
- providing connection information to a linked services module based on the portion of the metadata that corresponds to the respective one of the specified data sources;
- forming the connection to the respective one of the specified data sources via the linked services module;
- connecting, via the linked services module, to a secure storage that contains the associated access credentials for the respective one of the specified plurality of data sources;
- obtaining the access credentials for the respective one of the specified plurality of data sources from the secure storage by passing the key value to the secure storage via the linked services module;
- and
- using the access credentials for the respective one of the specified plurality of data sources in forming the connection to the respective one of the specified plurality of data sources via the linked services module;
- accessing, from the respective one of the specified plurality of data sources via the connection, a data object specified by the portion of the metadata that corresponds to the respective one of the specified plurality of data sources;
- and
- storing the data object in the data lake, the data lake being remote from each of the multiple data sources;
- creating a data source pipeline end log corresponding to an end of the sequence of activities for the respective one of the specified plurality of data sources when the data object is stored in the data lake;
- and
- create an overarching pipeline end log corresponding to an end of the overarching data pipeline after the sequence of activities has been performed for each of the specified plurality of data sources.