Data pipeline controller

A non-transitory computer-readable medium storing instructions which, when executed by a processing system including at least one processor when deployed in a communication network, cause the processing system to perform operations, the operations comprising:

• obtaining a request for a delivery of a data set to at least one destination;
• mapping the request to a first information model from among a plurality of information models, wherein the first information model comprises a flow sequence for a data pipeline;
• selecting a plurality of data schemas of a plurality of data pipeline component types from a catalog of data pipeline component types in accordance with the first information model, wherein the catalog of data pipeline component types comprises a first data schema and a first ontology for a first type of data pipeline component and a second data schema and a second ontology for a second type of data pipeline component, wherein the plurality of data schemas comprises the first data schema and the second data schema, and wherein the plurality of data pipeline component types comprises the first type of data pipeline component, and the second type of data pipeline component;
• determining a plurality of data pipeline components, wherein the plurality of data pipeline components comprises a plurality of host devices deployed in the communication network;
• and
• configuring the plurality of data pipeline components in accordance with the plurality of data schemas into the data pipeline, wherein the configuring comprises transmitting one or more commands over the communication network to one or more respective data pipeline components of the plurality of data pipeline components as defined in one or more respective data schemas of the plurality of data schemas associated with the one or more respective data pipeline components, wherein the data pipeline is to deliver the data set to the at least one destination, and wherein at least one of the one or more commands is to configure at least one of the plurality of data pipeline components as a data storage node.

An apparatus comprising:

• a processing system including at least one processor;
• and
• a computer-readable medium storing instructions which, when executed by the processing system when deployed in a communication network, cause the processing system to perform operations, the operations comprising:
• obtaining a request for a delivery of a data set to at least one destination;
• mapping the request to a first information model from among a plurality of information models, wherein the first information model comprises a flow sequence for a data pipeline;
• selecting a plurality of data schemas of a plurality of data pipeline component types from a catalog of data pipeline component types in accordance with the first information model, wherein the catalog of data pipeline component types comprises a first data schema and a first ontology for a first type of data pipeline component and a second data schema and a second ontology for a second type of data pipeline component, wherein the plurality of data schemas comprises the first data schema and the second data schema, and wherein the plurality of data pipeline component types comprises the first type of data pipeline component, and the second type of data pipeline component;
• determining a plurality of data pipeline components, wherein the plurality of data pipeline components comprises a plurality of host devices deployed in the communication network;
• and
• configuring the plurality of data pipeline components in accordance with the plurality of data schemas into the data pipeline, wherein the configuring comprises transmitting one or more commands over the communication network to one or more respective data pipeline components of the plurality of data pipeline components as defined in one or more respective data schemas of the plurality of data schemas associated with the one or more respective data pipeline components, wherein the data pipeline is to deliver the data set to the at least one destination, and wherein at least one of the one or more commands is to configure at least one of the plurality of data pipeline components as a data storage node.