A non-transitory computer-readable medium storing a computer program that, when executed by a processor, causes the processor to perform method operating computer-implemented system for inferred lineage wherein the method comprises:
- generating, by variation generation module of a transformation module of a processing subsystem, a plurality of variants by applying a plurality of transformations to each column of a plurality of source tables and corresponding target tables;
- using, by a comparison module of the transformation module of a processing subsystem, one or more column-similarity functions to compute a table similarity and compute an inverse document frequency term in a distributed manner;
- computing, by the comparison module of the transformation module of a processing subsystem, a term frequency in a distributed manner to determine a table lineage from a plurality of similarity scores;
- computing, by the comparison module of the transformation module of a processing subsystem a sparse pairwise similarity matrix for each factor of a plurality of factors of a column by formulating the term frequency and the inverse document frequency term;
- computing, by the comparison module of the transformation module of a processing subsystem, the sparse pairwise similarity matrix for each factor and a weighted sum of computed sparse pairwise similarity metrics to obtain a final sparse pairwise similarity matrix;
- sorting, by a sorting module of the transformation module of the processing subsystem, columns of the source table in descending order based on entries covered in a destination table and based on a column of the source table;
- revising, by the sorting module of the transformation module of the processing subsystem, a plurality of estimates covered by the destination table, wherein the revision is based on collection of tables to provide more weight to columns of source table based on destination table;
- pruning, by the sorting module of the transformation module of the processing subsystem, the column of the source table for a low revised estimates of coverage and remove remaining corresponding column lineages to maintain similarities among the columns that have a table lineage;
- calculating, by a similar descriptors transformation module of the processing subsystem, a histogram-based similarity of two columns and variants of columns if the column is non-numeric type;
- mapping, by the similar descriptors transformation module of the processing subsystem, a plurality of entries of the columns to count integers by sorting relative frequency and build a histogram for each column;
- calculating, by the similar descriptors transformation module of the processing subsystem, a score between the two histograms using a Jensen Shannon divergence method;
- computing, by the similar descriptors transformation module of the processing subsystem, an artificial intelligence embedding by calculating an embedding for each column, variants of columns, and computing similarities among the variants of columns by using a number of metrics;
- building, by a mutual information module of the processing subsystem, a supervised regression model to be fitted with a numerical source column as an input data and a numerical target column and record a loss value;
- ranking, by the mutual information module of the processing subsystem, importance of each feature corresponding to each numerical source column based on the score after fitting the supervised regression model;
- iteratively removing, by the mutual information module of the processing subsystem, one feature from the numerical source columns and re-fit the supervised regression model with new numerical source columns and the numerical target column by removing one feature and record a loss value;
- and
- stopping, by the mutual information module of the processing subsystem, the iterative removal of the feature at an occurrence a jump in loss value and generate a target column by using remaining features as top source columns, wherein the remaining features are the features remained after stopping the iterative removal of the feature.