Methods, apparatuses, and systems for ingesting and consuming data utilizing a trading partner manager

An apparatus for ingesting, enriching, and enabling the consumption of data utilizing a trading partner manager (TPM) interface, the apparatus comprising at least one processor and at least one memory including program code, the at least one memory and the program code configured to, with the processor, cause the apparatus to at least:

• provide the TPM interface for display via a user computing device, wherein (a) the TPM interface comprises a plurality of selectable predefined data fabric configurations selectable by a user operating the user computing device, and (b) the plurality of selectable predefined data fabric configurations are used to define a set of data fabric configurations;
• receive user input originating from the TPM interface, wherein (a) the user input comprises one or more selections of the plurality of selectable predefined data fabric configurations and defines the set of data fabric configurations, and (b) the set of data fabric configurations comprises a data source and a data source type;
• verify the one or more selections of the plurality of selectable predefined data fabric configurations;
• receive, into the data landing zone and based at least in part on the set of data fabric configurations, a data set from the data source in a batch process;
• ingest, based at least in part on the data fabric configurations, the received data set, wherein (a) ingesting the received data set comprises performing one or more pre-ingestion checks on each of the received data set, (b) the one or more pre-ingestion checks comprise one or more of a file count check, a duplicate file check, an entity count check, a file transfer completeness check, a schema evolution, or a timely arrival check;
• after performing the one or more pre-ingestion checks, store the ingested data set in a data lake, wherein the ingested data set is stored in its ingested form;
• after storing the ingested data set in the data lake, performing one or more post-ingestion checks on the ingested data set, wherein the one or more post-ingestion checks comprise one or more of a schema validation or a record count check;
• after performing the one or more post-ingestion checks, enrich, based at least in part on the data fabric configurations, the ingested data set, wherein enriching the ingested data set comprises one or more of generating a snapshot, generating a data quality report, performing identity matching solutions, performing address standardization, or generating a set of data quality timeliness measurements;
• determine, and based at least in part on the enriched data set, whether to use a hive view of the enriched data set or a snapshot view of the enriched data set;
• and
• in response to determining to use the snapshot view of the enriched data set, enable consumption of the enriched data set.

A non-transitory computer-readable storage medium for ingesting, enriching, and enabling the consumption of data utilizing a trading partner manager (TPM) interface, the non-transitory computer-readable storage medium storing program code instructions that, when executed, cause a data fabric computing device to:

• provide the TPM interface for display via a user computing device, wherein (a) the TPM interface comprises a plurality of selectable predefined data fabric configurations selectable by a user operating the user computing device, and (b) the plurality of selectable predefined data fabric configurations are used to define a set of data fabric configurations
• receive user input originating from the TPM interface, wherein (a) the user input comprises one or more selections of the plurality of selectable predefined data fabric configurations and defines the set of data fabric configurations, and (b) the set of data fabric configurations comprises a data source and a data source type;
• verify the one or more selections of the plurality of selectable predefined data fabric configurations;
• receive, into the data landing zone and based at least in part on the set of data fabric configurations, a data set from the data source in a batch process;
• ingest, based at least in part on the data fabric configurations, the received data set, wherein (a) ingesting the received data set comprises performing one or more pre-ingestion checks on each of the received data set, (b) the one or more pre-ingestion checks comprise one or more of a file count check, a duplicate file check, an entity count check, a file transfer completeness check, a schema evolution, or a timely arrival check;
• after performing the one or more pre-ingestion checks, store the ingested data set in a data lake, wherein the ingested data set is stored in its ingested form;
• after storing the ingested data set in the data lake, performing one or more post-ingestion checks on the ingested data set, wherein the one or more post-ingestion checks comprise one or more of a schema validation or a record count check;
• after performing the one or more post-ingestion checks, enrich, based at least in part on the data fabric configurations, the ingested data set, wherein enriching the ingested data set comprises one or more of generating a snapshot, generating a data quality report, performing identity matching solutions, performing address standardization, or generating a set of data quality timeliness measurements;
• determine, and based at least in part on the enriched data set, whether to use a hive view of the enriched data set or a snapshot view of the enriched data set;
• and
• in response to determining to use the snapshot view of the enriched data set, enable consumption of the enriched data set.