Method and system for selecting diversified data from a dataset

A system, comprising:

• a memory;
• and
• a processor configured to:
• determine numerical data and categorical data from the dataset, wherein the numerical data and the categorical data are continuously monitored and adjusted to ensure compliance with policies and regulations;
• identify, by the processor, a subset of uncorrelated numerical data based on correlations within the numerical data;
• remove, by the processor, the numerical data that exceed a predefined correlation threshold:
• prioritize, by the processor, the remaining numerical data based on a variability score for each numerical data feature;
• compute, by the processor, an importance score for each of categorical data feature from one or more categorical data features based on an entropy;
• arrange, by the processor, the one or more categorical data features based on a respective importance score;
• select a subset of high-importance categorical data features from the arranged one or more categorical data features;
• formulate a correlation between the selected numerical data and the categorical data feature;
• prepare a subset of data comprising the uncorrelated numerical data and the categorical data;
• allocate the subset of data into one or more risk groups based on predefined risk factors including at least one of geographical location, user identity, and number of identical transactions;
• assign a sample size to the one or more risk groups;
• choose samples of data from a risk group of the one or more risk groups by selecting an initial data point of the risk group and iteratively selecting a subsequent data point of the risk group based on angular and Euclidean distances of the initial data point from the subsequent data point of the risk group;
• generate a final dataset corresponding to the initial data point of the risk group and the subsequent data point of the risk group representing diversified variations with high coverage of both the numerical data and the categorical data;
• and
• store the final dataset to train an AI model for detecting deviations in the dataset.

A non-transitory computer-readable medium having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to execute a method for selecting diversified data from a dataset, comprising:

• determining numerical data and categorical data from the dataset, wherein the numerical data and the categorical data are continuously monitored and adjusted to ensure compliance with policies and regulations;
• identifying, by the processor, a subset of uncorrelated numerical data based on correlations within the numerical data;
• removing, by the processor, the numerical data that exceed a predefined correlation threshold;
• prioritizing, by the processor, the remaining numerical data based on a variability score for each numerical data feature;
• computing, by the processor, an importance score for each of categorical data feature from one or more categorical data features based on an entropy;
• arranging, by the processor, the one or more categorical data features based on a respective importance score;
• selecting a subset of high-importance categorical data features from the arranged one or more categorical data features;
• formulating a correlation between the selected numerical data and the categorical data feature;
• preparing a subset of data comprising the uncorrelated numerical data and the categorical data;
• allocating the subset of data into one or more risk groups based on predefined risk factors including at least one of geographical location, user identity, and number of identical transactions;
• assigning a sample size to the one or more risk groups;
• choosing samples of data from a risk group of the one or more risk groups by selecting an initial data point of the risk group and iteratively selecting a subsequent data point of the risk group based on angular and Euclidean distances of the initial data point from the subsequent data point of the risk group;
• generating a final dataset corresponding to the initial data point of the risk group and the subsequent data point of the risk group representing diversified variations with high coverage of both the numerical data and the categorical data;
• storing the final dataset to train an AI model for detecting deviations in the dataset.