System and method for file management in data structures

A method comprising:

• extracting, by at least one processor, one or more column names from a selected file stored to a database operable to store a plurality of files, wherein each column name identifies a column within the selected file;
• extracting, by the at least one processor, one or more file fields comprising data stored in one or more fields of the selected file;
• extracting, by the at least one processor, one or more terms identifying the selected file in the database;
• extracting metadata identifying source information of the one or more extracted column names, file fields, and terms;
• and
• associating one or more of the extracted column names and extracted file fields with one or more of the extracted terms to produce associated data;
• identifying one or more instances of unassociated data by determining that:
• an extracted term does not correspond to at least one of the extracted column names or at least one of the extracted file fields;
• an extracted column name does not correspond to at least one of the extracted terms or at least one of the extracted file fields;
• or
• a file field does not correspond to at least one of the extracted terms or the extracted column names;
• determining an end-to-end dataflow of the associated data by identifying one or more sources of the extracted columns, file fields, and terms from the extracted metadata;
• loading the associated data into the memory wherein loading data into the memory comprises performing either:
• a full load comprising loading all the associated data onto the memory;
• or
• a delta load comprising comparing the extracted database columns, file fields, and terms with data stored in the memory and loading any different data from the extracted data onto the memory;
• generating a journal recording metadata associated with the full load or the delta load;
• displaying the end-to-end dataflow of the associated data on a user interface;
• and
• transmitting a communication identifying the one or more instances of unassociated data.

One or more computer readable non-transitory storage media embodying software that is operable when executed by at least one processor to:

• extract one or more column names from a selected file stored to a database operable to store a plurality of files, wherein each column name identifies a column within the selected file;
• extract one or more file fields comprising data stored in one or more fields of the selected file;
• extract one or more terms identifying the selected file in the database;
• extract metadata identifying source information of the one or more extracted column names, file fields, and terms;
• and
• associate one or more of the extracted column names and extracted file fields with one or more of the extracted terms to produce associated data;
• identify one or more instances of unassociated data by determining that:
• an extracted term does not correspond to at least one or the extracted column names or at least one of the extracted file fields;
• an extracted column name does not correspond to at least one of the extracted terms or at least one of the extracted file fields;
• or
• a file field does not correspond to at least one or the extracted terms or the extracted column names;
• determine an end-to-end dataflow of the associated data by identifying one or more sources of the extracted columns, file fields, and terms from the extracted metadata;
• load the associated data into the memory wherein loading data into the memory comprises comparing the extracted database columns, file fields, and terms with data stored in the memory and loading any different data from the extracted data onto the memory;
• display the end-to-end dataflow of the associated data on a user interface;
• and
• transmit a communication identifying the one or more instances of unassociated data.