Condenser framework

A distributed computing system operating in a data lake and comprising processors linked through a communications network that execute machine executable instructions stored on non-transitory memory within the distributed computer system for generating structured views of data stored within the data lake, the system comprising:

• an edge-node positioned within the data lake and configured to receive tabulated data from computer systems outside the data lake in native format, wherein the native format corresponds to the tabulated data in its as-generated form;
• a view framework configured to:
• eliminate duplicate records included in the tabulated data;
• store unique records included in the tabulated data in a main flat table within the data lake;
• generate a first standard view table referencing a first set of unique records added to the main flat table within a first predetermined time window;
• and
• create a second standard view table referencing a second set of unique records added to the main flat table within a second predetermined time window;
• a condenser framework configured to improve performance of the view framework when generating the standard view table by:
• obtaining a count of the first set of unique records referenced in the first standard view table;
• creating a temporary table based on the count;
• inserting a set of overwrite records into the temporary table;
• and
• overwriting the main flat table with the temporary table;
• and
• a viewing module configured to generate views of:
• the first set of unique records referenced in the first standard view table by using metadata included in the first standard view table to locate one or more of the unique records included in the main flat table and stored within the data lake;
• and
• the second set of unique records referenced in the second standard view table by using metadata included in the second standard view table to locate one or more of the unique records included in the main flat table and stored within the data lake.

A distributed computer system comprising processors linked through a communications network that execute software applications stored on non-transitory memory within the distributed computer system and generate structured views of data stored in native format within a data lake, wherein the native format corresponds to the data in its as-generated form, the software applications comprising:

• a view framework configured to generate a first standard view table by removing duplicate records added to a main flat table stored in the data lake within a first predetermined time window;
• a condenser framework configured to improve performance of the view framework when generating the first standard view table by:
• obtaining a unique record count of records in the first standard view table;
• creating a temporary table based on the record count;
• inserting a set of overwrite records into the temporary table;
• overwriting the main flat table with the temporary table;
• and
• creating a second standard view table referencing unique records added to the main flat table within a second predetermined time window;
• and
• a viewing module configured to generate, based on the data stored in native format within the data and in response to user commands, a display of:
• the first standard view table during the first time window;
• and
• the second standard view table during the second time window.