In-querying data cleansing with semantic standardization

A system comprising:

• a processor;
• and a memory containing program code which when executed by the processor is configured to perform an operation, comprising:
• integrating a set of standardization functions into database engines to allow access to standardization of data via database queries executed in the database engines against databases;
• storing a plurality of standardization tables;
• with an extract module, extracting data before a transform portion of an extract, transform, and load process by issuing a first database query from the database queries to a first database engine of the database engines and a second database query from the database queries to a second database engine of the database engines, wherein the first database query includes a first standardization function, and wherein the second database query includes a second standardization function;
• with the first database engine,
• determining a first context included in the first standardization function;
• and
• invoking the first standardization function within the first database engine to convert a first data value having the first context using a first standard value and a first standardization table from the plurality of standardization tables that includes a context column for the first context;
• with the second database engine,
• determining a second context using metadata values by:
• identifying the metadata values of a column name and a database schema;
• mapping the metadata values to the second context using a lookup table;
• and
• identifying a second standardization table from the plurality of standardization tables based on the second context, wherein the second standardization table does not include any context column;
• and
• invoking the second standardization function within the second database engine to convert a second data value having the second context using a second standard value and the second standardization table;
• and providing cleansed data in the databases before another transform portion of another extract, transform, and load process by storing the converted first data value and the converted second data value.

A computer program product comprising a non-transitory computer readable storage medium storing computer readable program code, which when executed by a computer, causes the computer to:

• integrate a set of standardization functions into database engines to allow access to standardization of data via database queries executed in the database engines against databases;
• store a plurality of standardization tables;
• with an extract module, extract data before a transform portion of an extract, transform, and load process by issuing a first database query from the database queries to a first database engine of the database engines and a second database query from the database queries to a second database engine of the database engines, wherein the first database query includes a first standardization function, and wherein the second database query includes a second standardization function;
• with the first database engine,
• determine a first context included in the first standardization function;
• and
• invoke the first standardization function within the first database engine to convert a first data value having the first context using a first standard value and a first standardization table from the plurality of standardization tables that includes a context column for the first context;
• with the second database engine,
• determining determine a second context using metadata values by:
• identifying the metadata values of a column name and a database schema;
• mapping the metadata values to the second context using a lookup table;
• and
• identifying a second standardization table from the plurality of standardization tables based on the second context, wherein the second standardization table does not include any context column;
• and
• invoke the second standardization function within the second database engine to convert a second data value having the second context using a second standard value and the second standardization table;
• and
• provide cleansed data in the databases before another transform portion of another extract, transform, and load process by storing the converted first data value and the converted second data value.