Method and apparatus for creating shared pipelines for data processing

An apparatus for updating pipeline operations for data processing, the apparatus comprising at least one processor and at least one non-transitory memory including computer program code instructions, the computer program code instructions configured to, when executed, cause the apparatus to:

• receive pipeline information for at least one of a plurality of pipelines, wherein the pipeline information includes at least one of an input dataset, output dataset, input model, intermediate model, or output model, wherein each of the plurality of pipelines have a distinct output dataset;
• determine one or more of the plurality of pipelines to update based on similarities with the pipeline information received for at least one of the plurality of pipelines;
• update the one or more of the plurality of pipelines based on the pipeline information received, wherein updating the pipeline comprises updating at least one of the input model, intermediate model, or output model;
• and
• store the one or more updated pipelines.

A computer program product comprising at least one non-transitory computer-readable storage medium having computer-executable program code portions stored therein, the computer-executable program code portions comprising program code instructions configured to:

• receive pipeline information for at least one of a plurality of pipelines, wherein the pipeline information includes at least one of an input dataset, output dataset, input model, intermediate model, or output model, wherein each of the plurality of pipelines have a distinct output dataset;
• determine one or more of the plurality of pipelines to update based on similarities with the pipeline information received for at least one of the plurality of pipelines;
• update the one or more of the plurality of pipelines based on the pipeline information received, wherein updating the pipeline comprises updating at least one of the input model, intermediate model, or output model;
• and
• store the one or more updated pipelines.