Systems and methods for management of data platforms

A computer-implemented method for facilitating efficient analysis of data on a data platform, the method comprising performing by a processor the steps of:

• accessing via a file system a plurality of files stored on a non-transient storage medium, each file having a timestamp, and identifying a plurality of different candidate pairs among the plurality of files, each candidate pair comprising a respective first file of the plurality of files and a respective second file of the plurality of files, wherein the processor determines that the second file was created after the first file was created, based on the respective timestamps thereof;
• querying the file system by the processor and obtaining for each file in each candidate pair one or more file attributes selected from the group consisting of:
• date of file creation, time of file creation, update date of file creation, update time of file creation, file size, file name, and file path comprising one of more directories containing the file;
• calculating one or more respective scores for each of the candidate pairs using the one or more file attributes, each respective score being a measure of an extent to which the second file of the pair was derived from the first file of the pair, the one or more respective scores comprising at least one of a name-based score, a path-based score, a timestamp-based score, a data-based lineage score, a schema-based lineage score, and a format-based score;
• selecting a second file of the candidate pairs wherein none of the other files of the candidate pairs were derived from the selected second file based on the respective scores of the candidate pairs, wherein the score of a candidate pair is based at least in part on at least one of:
• (i) an overlap between contents of two files in the candidate pair, and (ii) an overlap between schemas of the two files in the candidate pair;
• identifying one or more ancestors of the selected second file based on the respective scores of the candidate pairs wherein each ancestor was used to derive the selected second file or was used to derive another ancestor of the selected second file;
• and
• generating a lineage for the selected second file wherein the lineage comprises the identified ancestors of the selected second file,
• wherein generating the lineage increases system efficiency by facilitating file purpose discovery.