Scalable identification of duplicate datasets in heterogeneous datasets

A computer system, the computer system comprising:

• one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage medium, and program instructions stored on at least one of the one or more computer-readable tangible storage medium for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising:
• receiving a plurality of heterogeneous datasets;
• automatically comparing schema information and metadata within each of the received plurality of heterogeneous datasets to generate name-based similarity scores for each combination of datasets;
• automatically pruning the plurality of heterogeneous datasets by removing datasets having no similarities;
• automatically identifying clusters of similar datasets using the name-based similarity scores for each dataset and generating mapping graphs illustrating each cluster of similar datasets;
• automatically comparing data distribution information within each of the received plurality of heterogeneous datasets to generate a plurality of data distribution similarity scores for each heterogeneous dataset;
• automatically calculating an overall distance metric using the name-based similarity scores and plurality of data distribution similarity scores;
• and
• based on the calculated overall distance metric, automatically generating distance graphs, wherein the automatically generated distance graphs identify clusters of similar datasets and illustrate inferred lineage for the clusters of similar datasets.

A computer program product, the computer program product comprising:

• one or more computer-readable tangible storage medium and program instructions stored on at least one of the one or more computer-readable tangible storage medium, the program instructions executable by a processor capable of performing a method, the method comprising:
• receiving a plurality of heterogeneous datasets;
• automatically comparing schema information and metadata within each of the received plurality of heterogeneous datasets to generate name-based similarity scores for each combination of datasets;
• automatically pruning the plurality of heterogeneous datasets by removing datasets having no similarities;
• automatically identifying clusters of similar datasets using the name-based similarity scores for each dataset and generating mapping graphs illustrating each cluster of similar datasets;
• automatically comparing data distribution information within each of the received plurality of heterogeneous datasets to generate a plurality of data distribution similarity scores for each heterogeneous dataset;
• automatically calculating an overall distance metric using the name-based similarity scores and plurality of data distribution similarity scores;
• and
• based on the calculated overall distance metric, automatically generating distance graphs, wherein the automatically generated distance graphs identify clusters of similar datasets and illustrate inferred lineage for the clusters of similar datasets.