Serverless data lake indexing subsystem and application programming interface

A method comprising:

• receiving a query;
• generating a query plan for the query;
• searching a data lake for built indexes stored at a predetermined location that conforms to an index specification, each built index including respective metadata that conforms to the index specification;
• determining, based on the respective index metadata for each built index found, a first built index to optimize the query plan, the first built index comprising first index metadata of the respective index metadata describing a lineage of the first built index, the lineage comprising an operation log that indicates operations that have taken place on the built index and usable to determine a data source corresponding to the first built index has been updated;
• optimizing the query plan to use the first built index;
• and
• executing the optimized query plan to provide a query result.

A system comprising:

• a data lake storing datasets and indexes based on the datasets;
• at least one processor coupled to the data lake;
• and
• at least one memory that stores program code executable by the at least one processor to perform operations, the operations comprising:
• receiving a query;
• generating a query plan for the query;
• searching a data lake for built indexes stored at a predetermined location that conforms to an index specification, each built index including respective metadata that conforms to the index specification;
• determining, based on the respective index metadata for each built index found, a first built index to optimize the query plan, the first built index comprising first index metadata of the respective index metadata describing a lineage of the first built index, the lineage usable to determine a data source corresponding to the first built index has been updated, the lineage indicating transformations applied to the data source when the first built index was built;
• optimizing the query plan to use the first built index;
• and
• executing the optimized query plan to provide a query result.