System for transforming data into table format

A system for transforming data into a distributed table format for a table, the system comprising:

• one or more hardware processors;
• and
• a memory storing instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising:
• accessing new data for a topic;
• generating data files based on the new data;
• generating materializer committables corresponding to the data files, each materializer committable comprising a metadata object describing one or more corresponding data files;
• performing one or more maintenance operations on previously committed files;
• generating a maintainer committable for each maintenance operation performed, each maintainer committable comprising a metadata object describing the corresponding maintenance operation;
• scheduling the maintainer and materializer committables for a commit process;
• and
• performing the commit process based on the scheduling, the commit process committing the maintainer and materializer committables to the distributed table format.

A machine-storage medium comprising instructions which, when executed by one or more processors of a machine, cause the machine to perform operations for transforming data into a distributed table format for a table, the operations comprising,

• accessing new data for a topic;
• generating data files based on the new data;
• generating materializer committables corresponding to the data files, each materializer committable comprising a metadata object describing one or more corresponding data files;
• performing one or more maintenance operations on previously committed files;
• generating a maintainer committable for each maintenance operation performed, each maintainer committable comprising a metadata object describing the corresponding maintenance operation;
• scheduling the maintainer and materializer committables for a commit process;
• and
• performing the commit process based on the scheduling, the commit process committing the maintainer and materializer committables to the distributed table format.