Automated data integration pipeline with storage and enrichment

A system comprising at least one computing device comprising at least one processor, a memory, and instructions executable by the at least one processor to create an application comprising:

• a) a data source discovery software module identifying a plurality of data sources associated with an automation environment;
• b) a data extraction software module retrieving data from at least one of the identified data sources, wherein the data is associated with a timestamp;
• c) a data mapping software module applying a first algorithm to map the retrieved data to a predetermined ontology, wherein the mapped data remains associated with the timestamp;
• d) a data storage software module storing timeseries of the mapped data in a data store;
• e) a data merging software module merging the mapped data into the data store, wherein the data store comprises a graph database, wherein each vertex in the graph includes a timeseries store to capture data changes over time based on timestamp associated with the mapped data, wherein the data merging software module is configured to merge the mapped data into the data store comprises matching mapped data with evolved vertices in the graph and merging new properties, shape details, or relationships into the matched vertices and timeseries data recorded in the vertex's timeseries store;
• f) a first data enrichment software module applying a second algorithm to identify patterns in the merged data and enrich the data based on one or more identified patterns, wherein the identified patterns are indicative of a relationship between one or more of the plurality data sources, wherein the relationship comprises a complete causal relationship, an incomplete causal relationship, or a correlational relationship, wherein enriching the data based on the one or more identified patterns comprises requesting creation or deletion of one or more vertices in the graph, one or more edges in the graph, one or more vertex properties in the graph, or one or more edge properties in the graph, wherein the enriching the data is performed by a plurality of software agents, each configured to generate a specific enrichment;
• g) a second data enrichment software module merging the enriched data into the data store to generate updated data and applying the second algorithm to further identify patterns in the updated data, wherein the enrichments are merged into the data store by origin tagging, wherein the origin tagging identifies the software agent that generated the enrichment;
• and
• h) a data access software module providing one or more APIs or one or more real-time streams comprising the enriched data, the updated data, or both the enriched data and the updated data.

A non-transitory computer-readable storage media encoded with instructions executable by one or more processors to provide an application comprising:

• a) a data source discovery module identifying a plurality of data sources associated with an automation environment;
• b) a data extraction module retrieving data from at least one of the identified data sources, wherein the data is associated with a timestamp;
• c) a data mapping module applying a first algorithm to map the retrieved data to a predetermined ontology, wherein the mapped data remains associated with the timestamp;
• d) a data storage module storing timeseries of the mapped data in a data store;
• e) a data merging module merging the mapped data into the data store, wherein the data store comprises a graph database, wherein each vertex in the graph includes a timeseries store to capture data changes over time based on the timestamp associated with the mapped data, wherein the data merging software module is configured to merge the mapped data into the data store comprises matching mapped data with evolved vertices in the graph and merging new properties, shape details, or relationships into the matched vertices and timeseries data recorded in the vertex's timeseries store;
• f) a first data enrichment module configured to apply a second algorithm to identify patterns in the merged data and enrich the data based on one or more identified patterns, wherein the identified patterns are indicative of a relationship between one or more of the plurality data sources, wherein the relationship comprises a complete causal relationship, an incomplete causal relationship, or a correlational relationship, wherein enriching the data based on the one or more identified patterns comprises requesting creation or deletion of one or more vertices in the graph, one or more edges in the graph, one or more vertex properties in the graph, or one or more edge properties in the graph, wherein the enriching the data is performed by a plurality of software agents, each configured to generate a specific enrichment;
• g) a second data enrichment module configured to merge the enriched data into the data store to generate updated data and apply the second algorithm to further identify patterns in the updated data, wherein the enrichments are merged into the data store by origin tagging, wherein the origin tagging identifies the software agent that generated the enrichment;
• and
• h) a data access module providing one or more APIs or one or more real-time streams comprising the enriched data, the updated data, or both the enriched data and the updated data.