Method and system for performing real-time data validation

A system for performing real-time data validation, the system comprising:

• a data validation device comprising a processor and a memory communicatively coupled to the processor, wherein the memory stores processor-executable instructions, which, on execution, causes the processor to:
• fetch a plurality of metadata from a set of external data sources and a set of external file storage systems, wherein the plurality of metadata is fetched along with a set of connection configuration parameters and column mapping information;
• verify the set of connection configuration parameters using a data validation scenario, wherein the set of connection configuration parameters comprises configuration details of a source connection and a target connection corresponding to each of the plurality of metadata, wherein the data validation scenario comprises performing data validation corresponding to each of a plurality of data fields with the column mapping information including both the source connection and the target connection and comparing existence of each of the plurality of data fields, and wherein the set of connection configuration parameters is verified by forming a connection string based on a type of connection;
• determine whether the verification of the set of connection configuration parameters is successful;
• storing the set of connection configuration parameters in a memory when the verification of the set of connection configuration parameters is success, rendering a message to a user to check the set of configuration parameters when the verification of the set of connection configuration parameters is unsuccessful;
• determine a number of clusters and a size of each cluster that are required for performing data validation based on an analysis of the plurality of metadata using an Artificial Intelligence (AI) model, wherein the AI model is trained for the external processing infrastructure to determine the number of clusters and the size of each of the set of clusters based on a size of data on the set of external data sources and the set of external file storage systems;
• generate a set of clusters for performing data validation in the external processing infrastructure based on the number of clusters and the size of each cluster;
• determine a set of test scripts from a plurality of test scripts based on the column mapping information and the set of connection configuration parameters, wherein the set of test scripts is configured to validate data from each of a plurality of external data sources and each of a plurality of external file storage systems;
• and
• perform a real-time data validation by executing the set of test scripts in the set of clusters.

A non-transitory computer-readable storage medium storing computer-executable instructions for:

• fetching a plurality of metadata from a set of external data sources and a set of external file storage systems, wherein the plurality of metadata is fetched along with a set of connection configuration parameters and column mapping information;
• verify the set of connection configuration parameters using a data validation scenario, wherein the set of connection configuration parameters comprises configuration details of a source connection and a target connection corresponding to each of the plurality of metadata, wherein the data validation scenario comprises performing data validation corresponding to each of a plurality of data fields with the column mapping information including both the source connection and the target connection and comparing existence of each of the plurality of data fields, and wherein the set of connection configuration parameters is verified by forming a connection string based on a type of connection;
• determining whether the verification of the set of connection configuration parameters is successful;
• storing the set of connection configuration parameters in a memory when the verification of the set of connection configuration parameters is success, rendering a message to a user to check the set of configuration parameters when the verification of the set of connection configuration parameters is unsuccessful;
• determining a number of clusters and a size of each cluster that are required for performing data validation based on an analysis of the plurality of metadata using an Artificial Intelligence (AI) model, wherein the AI model is trained for the external processing infrastructure to determine the number of clusters and the size of each of the set of clusters based on a size of data on the set of external data sources and the set of external file storage systems;
• generating a set of clusters for performing data validation in the external processing infrastructure based on the number of clusters and the size of each cluster;
• determining a set of test scripts from a plurality of test scripts based on the column mapping information and the set of connection configuration parameters, wherein the set of test scripts is configured to validate data from each of a plurality of external data sources and each of a plurality of external file storage systems;
• and
• performing a real-time data validation by executing the set of test scripts in the set of clusters.