Real-time data quality assurance by integrating ai-driven anomaly detection, explainable ai, and automated root cause analysis for superior data quality in the etl pipelines

A system comprising:

• at least one processing device comprising a processor coupled to a memory;
• the at least one processing device being configured:
• to receive, by an Extract, Transform, Load (ETL) anomaly detection system, source data for processing through an ETL pipeline;
• to detect, by the ETL anomaly detection system, anomalies in the source data using a trained machine learning model, wherein the trained machine learning model comprises a Random Forest classifier with multiple decision trees, a feature engineering pipeline that calculates average prices by product model and price deviations, a probability threshold of 0.5 for anomaly classification, and risk level categorization based on probability scores above 0.75 for high risk and between 0.5-0.75 for medium risk;
• to analyze, by the ETL anomaly detection system, the detected anomalies using Shapley Additive Explanations (SHAP) values to determine contributing factors to the anomalies;
• to validate, by the ETL anomaly detection system, quality of the source data based on the detected anomalies and SHAP analysis, wherein validating the quality of the source data based on the detected anomalies and SHAP analysis occurs before the source data reaches the target system;
• and
• to load, by the ETL anomaly detection system, validated data into a target system while maintaining application continuity, and wherein, when anomalies are detected, data is quarantined rather than loaded into the target system.

A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

• to receive, by an Extract, Transform, Load (ETL) anomaly detection system, source data for processing through an ETL pipeline;
• to detect, by the ETL anomaly detection system, anomalies in the source data using a trained machine learning model, wherein the trained machine learning model comprises a Random Forest classifier with multiple decision trees, a feature engineering pipeline that calculates average prices by product model and price deviations, a probability threshold of 0.5 for anomaly classification, and risk level categorization based on probability scores above 0.75 for high risk and between 0.5-0.75 for medium risk;
• to analyze, by the ETL anomaly detection system, the detected anomalies using Shapley Additive Explanations (SHAP) values to determine contributing factors to the anomalies;
• to validate, by the ETL anomaly detection system, quality of the source data based on the detected anomalies and SHAP analysis, wherein validating the quality of the source data based on the detected anomalies and SHAP analysis occurs before the source data reaches the target system;
• and
• to load, by the ETL anomaly detection system, validated data into a target system while maintaining application continuity, and wherein, when anomalies are detected, data is quarantined rather than loaded into the target system.