Dataflow graph processing with expectations

A method, comprising:

• receiving, from a client device, an indication to generate an extract, transform and load (ETL) data processing pipeline defined with respect to a set of operations, the set of operations received via a communication interface of the client device and including one or more declarative structured query language (SQL) queries, each operation configured to read one or more input datasets and transform the one or more input datasets stored in a storage location of a cloud platform into one or more output datasets, at least one operation indicating an expectation describing one or more properties data in the input datasets or the output datasets of the at least one operation is expected to possess, wherein at least two operations are not in order;
• generating an in-memory dataflow graph for the data processing pipeline, comprising:
• determining data dependencies between the set of operations of the data processing pipeline;
• defining, based on the determined dependencies, an ordering in which the set of operations of the data processing pipeline is performed;
• and
• generating the dataflow graph reflecting a correct ordering of the set of operations, the dataflow graph including a set of nodes and edges between the nodes, an edge in the dataflow graph representing a dependency between a node representing an input dataset and another node representing an output dataset for a respective operation;
• testing the dataflow graph, comprising:
• obtaining a sample set of data and storing the sample set of data in a storage device;
• executing the at least one operation to the sample set of data;
• determining whether the execution of the at least one operation meets the expectation;
• and
• responsive to the expectation being met, determining that the dataflow graph is validated;
• processing the set of operations associated with the data processing pipeline such that the expectation for the at least one operation is enforced;
• and
• providing a result of the processing to the client device.

A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:

• receiving, from a client device, an indication to generate an extract, transform and load (ETL) data processing pipeline defined with respect to a set of operations, the set of operations received via a communication interface of the client device and including one or more declarative structured query language (SQL) queries, each operation configured to read one or more input datasets and transform the one or more input datasets stored in a storage location of a cloud platform into one or more output datasets, at least one operation indicating an expectation describing one or more properties data in the input datasets or the output datasets of the at least one operation is expected to possess, wherein at least two operations are not in order;
• generating an in-memory dataflow graph for the data processing pipeline, comprising:
• determining data dependencies between the set of operations of the data processing pipeline;
• defining, based on the determined dependencies, an ordering in which the set of operations of the data processing pipeline is performed;
• and
• generating the dataflow graph reflecting a correct ordering of the set of operations, the dataflow graph including a set of nodes and edges between the nodes, an edge in the dataflow graph representing a dependency between a node representing an input dataset and another node representing an output dataset for a respective operation;
• testing the dataflow graph, comprising:
• obtaining a sample set of data and storing the sample set of data in a storage device:
• executing the at least one operation to the sample set of data;
• determining whether the execution of the at least one operation meets the expectation;
• and
• responsive to the expectation being met, determining that the dataflow graph is validated;
• processing the set of operations associated with the data processing pipeline such that the expectation for the at least one operation is enforced;
• and
• providing a result of the processing to the client device.