Techniques for unifying ETL filter operators

A computer system, comprising

• one or more processors;
• a memory in communication with the one or more processors, the memory configured to store computer-executable instructions, wherein executing the computer-executable instructions causes the one or more processors to perform steps comprising:
• receiving an extract, transform, load (ETL) plan comprising a primary split operator and a plurality of primary filter operators, wherein a data pipeline of a plurality of primary data pipelines comprises a secondary split operator of a plurality of secondary data pipelines configured to act on data output by the primary split operator, and wherein the ETL plan further comprises the plurality of secondary data pipelines output by the secondary split operator;
• in response to the receiving the ETL plan:
• identifying the plurality of primary filter operators that are configured to act on the data output by the primary split operator in the extract, transform, load plan;
• automatically generating a primary unified filter operator using the plurality of primary filter operators;
• generating, based on the received ETL plan and the data pipeline, an updated extract, transform, load plan comprising the primary unified filter operator providing filtered data to the primary split operator;
• providing the updated ETL plan to a client device via a network;
• and
• storing the updated extract, transform, load plan in a data store of the client device.

A non-transitory computer-readable storage medium, storing computer-executable instructions that, when executed, cause one or more processors of a computer system to perform steps comprising:

• receiving, by a computer system, an extract, transform, load (ETL) plan comprising a primary split operator and a plurality of primary filter operators, wherein a data pipeline of a plurality of primary data pipelines comprises a secondary split operator of a plurality of secondary data pipelines configured to act on data output by the primary split operator, and wherein the ETL plan further comprises the plurality of secondary data pipelines output by the secondary split operator;
• in response to the receiving the ETL plan:
• identifying, by the computer system, the plurality of primary filter operators that are configured to act on the data output by the primary split operator in the extract, transform, load plan;
• automatically generating, by the computer system, a primary unified filter operator using the plurality of primary filter operators;
• generating, by the computer system based on the received ETL plan and the data pipeline, an updated extract, transform, load plan comprising the primary unified filter operator providing filtered data to the primary split operator;
• providing, by the computer system, the updated ETL plan to a client device via a network;
• and
• storing the updated extract, transform, load plan in a data store of the client device.