US 2025/0097290 A1Published application

Data pipeline orchestration for data-driven engineering

AssigneeSap SE

InventorsFabian ARNDT, Enes OERDEK

Life of this patent

Estimating the term…

Loading patent…

Abstract

A system associated with data pipeline orchestration may include a data pipeline data store that contains, for each of a plurality of data pipelines, a series of data pipeline steps associated with a data pipeline use case. A data pipeline orchestration server may receive, from a data engineering operator, a selection of a data pipeline use case in the data pipeline data store. The data pipeline orchestration server may also receive first configuration information for the selected data pipeline use case and second configuration information, different than the first configuration information, for the selected data pipeline use case. The data pipeline orchestration server may then store representations of both the first configuration information and the second configuration information in connection with the selected data pipeline use case. Execution of the selected pipeline is then arranged in accordance with one of the first configuration information and the second configuration information.

Claims20

Dependent claims

A system associated with data pipeline orchestration, comprising:

a data pipeline data store containing, for each of a plurality of data pipelines, a series of data pipeline steps associated with a data pipeline use case, wherein the series of data pipeline steps include:
automatically downloading raw data from an internal enterprise data source;
automatically performing extract, transform, load tasks;
automatically performing data cleanup;
and automatically storing information in a cloud-based data warehouse;
and
a data pipeline orchestration server, coupled to the data pipeline data store, including:
a computer processor, and
a computer memory storage device, coupled to the computer processor, that contains instructions that when executed by the computer processor enable the data pipeline orchestration server to:
(i) receive, from a data engineering operator, a selection of a data pipeline use case in the data pipeline data store,
(ii) receive first configuration information for the selected data pipeline use case,
(iii) receive second configuration information, different than the first configuration information, for the selected data pipeline use case,
(iv) store representations of both the first configuration information and the second configuration information in connection with the selected data pipeline use case, and
(v) arrange for an automatic execution of the selected pipeline in accordance with one of the first configuration information and the second configuration information,
wherein a data pipeline use case associated with one data engineering team of an enterprise is shared with another data engineering team of the enterprise via a platform and cloud-based service for software development and version control.

12 more elements

A computer-implemented method associated with data pipeline orchestration, comprising:

receiving, at a computer processor of a data pipeline orchestration server from a data engineering operator, a selection of a data pipeline use case in a data pipeline data store, wherein the data pipeline data store contains, for each of a plurality of data pipelines, a series of data pipeline steps associated with a data pipeline use case, wherein the series of data pipeline steps include:
automatically downloading raw data from an internal enterprise data source;
automatically performing extract, transform, load tasks;
automatically performing data cleanup;
and automatically storing information in a cloud-based data warehouse;
receiving first configuration information for the selected data pipeline use case;
receiving second configuration information, different than the first configuration information, for the selected data pipeline use case;
storing representations of both the first configuration information and the second configuration information in connection with the selected data pipeline use case;
and
arranging for an automatic execution of the selected pipeline in accordance with one of the first configuration information and the second configuration information,
wherein a data pipeline use case associated with one data engineering team of an enterprise is shared with another data engineering team of the enterprise via a platform and cloud-based service for software development and version control.

8 more elements

A non-transitory, computer readable medium having executable instructions stored therein to implement a method associated with data pipeline orchestration, the method comprising:

receiving, at a computer processor of a data pipeline orchestration server from a data engineering operator, a selection of a data pipeline use case in a data pipeline data store, wherein the data pipeline data store contains, for each of a plurality of data pipelines, a series of data pipeline steps associated with a data pipeline use case, wherein the series of data pipeline steps include:
automatically downloading raw data from an internal enterprise data source;
automatically performing extract, transform, load tasks;
automatically performing data cleanup;
and automatically storing information in a cloud-based data warehouse;
receiving first configuration information for the selected data pipeline use case;
receiving second configuration information, different than the first configuration information, for the selected data pipeline use case;
storing representations of both the first configuration information and the second configuration information in connection with the selected data pipeline use case;
and
arranging for an automatic execution of the selected pipeline in accordance with one of the first configuration information and the second configuration information,
wherein a data pipeline use case associated with one data engineering team of an enterprise is shared with another data engineering team of the enterprise via a platform and cloud-based service for software development and version control.

8 more elements

Dependencies are inferred from the claim text.