US 2025/0371024 A1 · DOMAIN SPECIFIC LANGUAGE FOR DATA PIPELINE CREATION

Abstract

A domain specific language, the “data pipeline language” (DPL), has been developed for programmatic creation of data pipelines for ETL/ELT with a human-readable format. The DPL syntax is abstracted from the specific technologies used for building a data pipeline, so the DPL that defines various components of a data pipeline (e.g., data sources, data warehouses, and/or cloud-based processing components) is independent of the specific vendors used to provide these components. A user that defines a data pipeline in the DPL specifies any external vendors that will be leveraged for execution of the data pipeline in a configuration file. During execution of a data pipeline defined in with the DPL, a pipeline execution platform maps data pipeline components identified from the DPL file to the respective external vendor(s) indicated in the configuration and “translates” the DPL components into their corresponding commands that are sent to the external vendor(s).

Claims20

Dependent claims

A method comprising:

obtaining a data pipeline file and a configuration file, wherein the data pipeline file comprises a definition of a data pipeline and indicates a plurality of components, wherein the data pipeline is an extract, transform, load (ETL) or an extract, load, transform (ELT) data pipeline;
and
executing the data pipeline, wherein executing the data pipeline comprises,
determining that a first component of the plurality of components corresponds to at least one of a data source, a data warehouse, and a processing component of a first external vendor;
and
determining a first command that corresponds to the first component and the first external vendor based on a type of the first component and the configuration file, wherein the configuration file indicates one or more external vendors to be used by the data pipeline;
and
communicating the first command to the first external vendor to perform one or more actions corresponding to execution of the first component of the data pipeline.

5 more elements

One or more non-transitory machine-readable media having program code stored thereon, the program code comprising instructions to:

obtain a data pipeline file and a configuration file, wherein the data pipeline file comprises a definition of a data pipeline and indicates a plurality of components, wherein the data pipeline is an extract, transform, load (ETL) or an extract, load, transform (ELT) data pipeline;
and
execute the data pipeline, wherein the instructions to execute the data pipeline comprise instructions to,
determine that a first component of the plurality of components corresponds to at least one of a data source, a data warehouse, and a processing component of a first external vendor;
and
determine a first command that corresponds to the first component and the first external vendor based on a type of the first component and the configuration file, wherein the configuration file indicates one or more external vendors to be used by the data pipeline;
and
communicate the first command to the first external vendor to perform one or more actions corresponding to execution of the first component of the data pipeline.

5 more elements

An apparatus comprising:

a processor;
and
a machine-readable medium having instructions stored thereon that are executable by the processor to cause the apparatus to,
obtain a data pipeline file and a configuration file, wherein the data pipeline file comprises a definition of a data pipeline and indicates a plurality of components, wherein the data pipeline is an extract, transform, load (ETL) or an extract, load, transform (ELT) data pipeline;
and
execute the data pipeline, wherein the instructions to execute the data pipeline comprise instructions to,
determine that a first component of the plurality of components corresponds to at least one of a data source, a data warehouse, and a processing component of a first external vendor;
and
determine a first command that corresponds to the first component and the first external vendor based on a type of the first component and the configuration file, wherein the configuration file indicates one or more external vendors to be used by the data pipeline;
and
communicate the first command to the first external vendor to perform one or more actions corresponding to execution of the first component of the data pipeline.

8 more elements

Dependencies are inferred from the claim text.

Domain specific language for data pipeline creation

Life of this patent

Domain specific language for data pipeline creation

Life of this patent

Abstract

Claims20