Methods and apparatuses for improved data ingestion using standardized plumbing fields

An apparatus for improved data ingestion in a relational database management system, the apparatus comprising at least one processor and at least one memory storing computer-executable instructions, that, when executed by the at least one processor, cause the apparatus to:

• ingest, using a set of standardized plumbing fields comprising descriptive fields including a stage table primary key field, a source primary key field, and a source attribute field, one or more records from a plurality of data sources into one or more stage tables in a persistent staging layer, wherein each of the one or more stage tables corresponds to one of the plurality of data sources,
• determine whether each particular record of the one or more records corresponds to any preexisting record of a plurality of preexisting records in the one or more stage tables based on:
• a hash of at least one descriptive field and a date/time associated with the particular record, wherein the hash of the at least one descriptive field and the date/time associated with the particular record uniquely identifies the particular record, and wherein the date/time associated with the particular record comprises at least one of a timestamp or a date that the particular record was ingested into the persistent staging layer, and
• a plurality of hashes of the at least one descriptive field and a date/time associated with each preexisting record of the plurality of preexisting records, wherein each hash of the plurality of hashes uniquely identifies one preexisting record of the plurality of preexisting records,
• and
• generate, using the one or more stage tables in the persistent staging layer, a data integration layer for modeling data represented by the data ingested into the one or more stage tables in the persistent staging layer.

A computer program product comprising at least one non-transitory computer-readable storage medium for improved data ingestion in a relational database management system, the at least one non-transitory computer-readable storage medium storing computer-executable instructions that, when executed, cause an apparatus to:

• ingest, using a set of standardized plumbing fields comprising descriptive fields including a stage table primary key field, a source primary key field, and a source attribute field, one or more records from a plurality of data sources into one or more stage tables in a persistent staging layer, wherein each of the one or more stage tables corresponds to one of the plurality of data sources,
• determine whether each particular record of the one or more records corresponds to any preexisting record of a plurality of preexisting records in the one or more stage tables based on:
• a hash of at least one descriptive field and a date/time associated with the particular record, wherein the hash of the at least one descriptive field and the date/time associated with the particular record uniquely identifies the particular record, and wherein the date/time associated with the particular record comprises at least one of a timestamp or a date that the particular record was ingested into the persistent staging layer, and
• a plurality of hashes of the at least one descriptive field and a date/time associated with each preexisting record of the plurality of preexisting records, wherein each hash of the plurality of hashes uniquely identifies one preexisting record of the plurality of preexisting records, and
• generate, using the one or more stage tables in the persistent staging layer, a data integration layer for modeling data represented by the data ingested into the one or more stage tables in the persistent staging layer.