Query generation and processing system

A method for processing queries on batch and streaming datasets, the method comprising:

• storing data ingested from one or more data sources in a first dataset and a second dataset;
• obtaining a data expression in a simplified query language;
• identifying a first quad corresponding to the first dataset and a second quad corresponding to the second dataset based on the data expression, wherein the data expression does not expressly identify a join between the first quad and the second quad;
• determining an implicit join between the first quad and the second quad based on an unambiguous relationship obtainable from a schema of the first dataset and the second dataset, including:
• generating a derivation graph including nodes representing the quads, the generating including producing a tree including leaves corresponding to the quads based on an arrangement of the quads within the data expression and recursively iterating through leaves of the tree to add nodes representing the quads to the derivation graph, and
• querying the derivation graph based on a grain of the quads;
• and
• generating a query in a structured query language based on the implicit join between the first quad and the second quad;
• and
• executing the query using a query pipeline that uses the data of the first dataset and the second dataset.

An apparatus for processing queries on batch and streaming datasets, the apparatus comprising:

• a memory storing instructions;
• and
• a processor configured to execute the instructions to:
• store data ingested from one or more data sources in a first dataset and a second dataset;
• obtain a data expression in a simplified query language;
• generate a query in a structured query language based on an implicit join determined between a first quad corresponding to the first dataset and a second quad corresponding to the second dataset based on an unambiguous relationship obtainable from a schema of the first dataset and the second dataset, wherein the data expression does not expressly identify a join between the first quad and the second quad;
• and
• execute the query using a query pipeline that uses the data of the first dataset and the second dataset,
• wherein the data stored in the first dataset and the second data set are stored directly into a memory buffer to enable compute nodes of a query pipeline that executes the query to access to the data without making a copy of the data and without waiting for the data to be stored in a cold storage,
• wherein the query pipeline includes one or more compute nodes instantiated based on a query plan for a query, wherein the one or more compute nodes include alternating layers of faucets and turbines, and wherein an upstream faucet transmits pointers to the ingested data within the memory buffer to one or more downstream turbines.