Flexible data ingestion

A computer-implemented system comprising:

• a storage, and
• a processor that receives data comprising a file comprising a database table, wherein the file is in a non-standard binary format and comprises a delta update to previously received data comprising a second file,
• stores the data comprising the file unaltered as a source data chunk in the storage;
• generates a processed data chunk from the source data chunk by converting the file to a standard binary format and storing the file in the processed data chunk without altering the source data chunk,
• generates a materialized data chunk from the processed data chunk by appending the processed data chunk to a second processed data chunk and performing, with a database server engine, at least one database operation on a database table of a file of the processed data chunk and the second processed data chunk and storing the file in the materialized data chunk without altering either the processed data chunk or the second processed data chunk, wherein the second processed data chunk was generated from a second source data chunk that comprises the previously received data and wherein the database table of the file of the processed data chunk and the second processed data chunk comprises the database table of the processed data chunk appended to a database table of the second processed data chunk;
• and
• makes the database table of the file of the materialized data chunk available for querying by the database server engine.

A system comprising:

• one or more computers and one or more storage devices storing instructions which are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
• comprising:
• receiving, at a computing device, data comprising a file comprising a database table, wherein the file is in a non-standard binary format and comprises a delta update to previously received data comprising a second file;
• storing the data comprising the file unaltered as a source data chunk;
• generating, by an ingestion engine of the computing device, a processed data chunk from the source data chunk by converting the file to a standard binary format and storing the file in the processed data chunk without altering the source data chunk;
• generating, by the ingestion engine of the computing device, a materialized data chunk from the processed data chunk by appending the processed data chunk to a second processed data chunk and performing, with a database server engine of the computing device, at least one database operation on a database table of a file of the processed data chunk and the second processed data chunk and storing the file in the materialized data chunk without altering either the processed data chunk or the second processed data chunk, wherein the second processed data chunk was generated from a second source data chunk that comprises the previously received data and wherein the database table of the file of the processed data chunk and the second processed data chunk comprises the database table of the processed data chunk appended to a database table of the second processed data chunk;
• and
• making the database table of the file of the materialized data chunk available for querying by the database server engine.