System and method for combinatorial data outlier detection via database query statement generation

A computer program product for combinatorial data outlier detection via database query statement generation, the computer program product comprising a non-transitory computer-readable medium comprising code configured to control a processing device to:

• receive a table comprising rows and columns, wherein each column comprises records and a corresponding data element;
• determine, based on each column and using a first machine learning model, a datatype to be assigned to each data element, wherein the datatype is selected from a list consisting of numeric, date, and text;
• generate conditions for columns corresponding to each data element, wherein conditions are selected based on the datatype, and wherein conditions are selected from a group consisting of null conditions, operator conditions, and value conditions;
• combine the conditions into a predetermined number of condition combinations, each condition combination unique from other condition combinations and comprising a combinatorial sequencing of the conditions having logical operators (AND, OR) between each condition such that the predetermined number of condition combinations comprises all possible logical arrangements of the conditions;
• generate a query statement for each of the condition combinations;
• query, in parallel with combining the conditions into the predetermined number of condition combinations, the table with each query statement to determine query results, wherein the query results match at least one of the condition combinations;
• determine, for each query statement, a data element quantity in the query results and a record quantity corresponding to each data element in the query results;
• tag the query statement as a rule in response to the data element quantity in the query results being within a first predetermined upper and lower limit, and the record quantity corresponding to each data element in the query results being within a second predetermined upper and lower limit, wherein the second predetermined upper and lower limit comprises thresholds expressed as percentages of total number of records within a column;
• extract records corresponding to the rule by using metadata associated with the rule to retrieve the records corresponding to the rule from the table;
• apply an outlier detection model to the records corresponding to the rule;
• and
• identify, using the outlier detection model, outliers in the records, wherein the outliers are detected based on proximity to clusters generated by the outlier detection model.

A computer-implemented method for combinatorial data outlier detection via database query statement generation, the method comprising:

• receiving a table comprising rows and columns, wherein each column comprises records and a corresponding data element;
• determining, based on each column and using a first machine learning model, a datatype to be assigned to each data element, wherein the datatype is selected from a list consisting of numeric, date, and text;
• generating conditions for columns corresponding to each data element, wherein conditions are selected based on the datatype, and wherein conditions are selected from a group consisting of null conditions, operator conditions, and value conditions;
• combining the conditions into a predetermined number of condition combinations, each condition combination unique from other condition combinations and comprising a combinatorial sequencing of the conditions having logical operators (AND, OR) between each condition such that the predetermined number of condition combinations comprises all possible logical arrangements of the conditions;
• generating a query statement for each of the condition combinations;
• querying, in parallel with combining the conditions into the predetermined number of condition combinations, the table with each query statement to determine query results, wherein the query results match at least one of the condition combinations;
• determining, for each query statement, a data element quantity in the query results and a record quantity corresponding to each data element in the query results;
• tag the query statement as a rule in response to the data element quantity in the query results being within a first predetermined upper and lower limit, and the record quantity corresponding to each data element in the query results being within a second predetermined upper and lower limit, wherein the second predetermined upper and lower limit comprises thresholds expressed as percentages of total number of records within a column;
• extract records corresponding to the rule by using metadata associated with the rule to retrieve the records corresponding to the rule from the table;
• apply an outlier detection model to the records corresponding to the rule;
• and
• identify, using the outlier detection model, outliers in the records, wherein the outliers are detected based on proximity to clusters generated by the outlier detection model.