Managed Tables for Data Lakes

A system comprising:

• one or more processors;
• and
• one or more storage devices coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for processing queries, the operations comprising:
• receiving a first request from a query engine to read one or more data files;
• selecting to read the one or more data files through the storage API;
• retrieving the one or more data files from either the read-optimized cloud storage or a write-optimized buffer using column-level metadata stored in an appendable distributed file system;
• reading the one or more data files in response to the request from the query engine;
• receiving a second request to read one or more additional data files;
• selecting to directly read the one or more additional data files from the read-optimized cloud storage;
• retrieving the one or more additional data files from the read-optimized cloud storage using a metadata snapshot of the column-level metadata;
• and
• reading the one or more additional data files in response to the second request.

A non-transitory computer readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for processing queries, the operations comprising:

• receiving a first request from a query engine to read one or more data files;
• selecting to read the one or more data files through the API;
• retrieving the one or more data files from either the read-optimized cloud storage or a write-optimized buffer using column-level metadata stored in an appendable distributed file system;
• reading the one or more data files in response to the request from the query engine;
• receiving a second request to read one or more additional data files;
• selecting to directly read the one or more additional data files from the read-optimized cloud storage;
• retrieving the one or more additional data files from the read-optimized cloud storage using a metadata snapshot of the column-level metadata;
• and
• reading the one or more additional data files in response to the second request.