Generating observability metrics for data lake usage based on data layer activity logs

A system comprising:

• a non-transitory storage medium storing computer program instructions;
• and
• a processor configured to execute the computer program instructions to cause operations comprising:
• generating an object path dataset comprising a plurality of rows of object paths for accessing corresponding data tables in a data lake;
• generating a table path dataset comprising a plurality of rows of table paths of physical addresses for the corresponding data tables in the data lake;
• expanding columns of the object path dataset to generate an expanded object path dataset by adding a plurality of additional columns, each additional column including object paths with a portion of information stripped away from corresponding object paths in a previous column;
• expanding rows of the table path dataset to generate an expanded table path dataset by adding an additional row for each original row, the additional row comprising a corresponding table path without partition information;
• iteratively matching the columns of the expanded object path dataset to the original rows and the additional rows;
• and
• generating observability metrics of access of the data tables based on the iterative matching.