Method and apparatus for modelling data exchange in a data flow of an extract, transform, and load (etl) process

A computer-implemented method for processing a data flow associated with an extract, transform, and load (ETL) process, the method comprising:

• receiving a data flow defining a sequence of operations for an ETL process;
• converting the data flow into a logical operator graph (LOG), the logical operator graph being a representation of the data flow and including a plurality of operators corresponding to the sequence of operations defined by the data flow;
• converting the logical operator graph into a query graph model (QGM) including automatically inserting a data station operator between a first operator and a second operator of the plurality of operators associated with the logical operator graph, the data station operator representing a staging point between the first operator and the second operator and being automatically inserted based on predetermined criteria;
• and
• generating an execution plan graph based on the query graph model, the execution plan graph including code to manage data staging between the first operator and the second operator.

A computer program product, tangibly stored on a computer-readable medium, for processing a data flow associated with an extract, transform, and load (ETL) process, the computer program product comprising instructions for causing a programmable processor to:

• receive a data flow defining a sequence of operations for an ETL process;
• convert the data flow into a logical operator graph (LOG), the logical operator graph being a representation of the data flow and including a plurality of operators corresponding to the sequence of operations defined by the data flow;
• convert the logical operator graph into a query graph model (QGM) including automatically inserting a data station operator between a first operator and a second operator of the plurality of operators associated with the logical operator graph, the data station operator representing a staging point between the first operator and the second operator and being automatically inserted based on pre-determined criteria;
• and
• generate an execution plan graph based on the query graph model, the execution plan graph including code to manage data staging between the first operator and the second operator.