Systems and methods for dynamic industrial data integration pipeline with components that automatically build data lineage and context as part of the pipeline execution

A non-transitory computer readable medium storing instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations comprising:

• deploying a data integration pipeline to an integration runtime environment executing in a container, wherein the container comprises a plurality of components, including a data context component and a data lineage component, and wherein the data integration pipeline is based on a pipeline template;
• providing, to the data integration pipeline, operational data generated by a data source during operation of an industrial automation system;
• receiving, from the data integration pipeline, context data for the operational data generated by the data context component, wherein the context data identifies the data source of the operational data;
• transmitting the context data to a data platform for storage;
• receiving, from the data integration pipeline, lineage data for the operational data generated by the data lineage component, wherein the lineage data comprises a chain of custody of the operational data from the data source to the data integration pipeline;
• and
• transmitting the lineage data to the data platform for storage.

A method, comprising:

• receiving, at a data integration pipeline, operational data generated by a data source during operation of an industrial automation system;
• generating context data for the operational data, wherein the context data identifies the data source of the operational data;
• generating lineage data for the operational data, wherein the lineage data comprises a chain of custody of the operational data from the data source to the data integration pipeline;
• and
• transmitting the operational data, the context data, and the lineage data for storage, processing, integration, display, or a combination thereof.