Closed-loop unified metadata architecture with universal metadata repository

A closed-loop unified metadata processing system comprising:

• a processor;
• universal metadata circuitry executable by the processor to retrieve source metadata from each of a plurality of different selected data sources that provide input data to a pre-defined workflow identified for analysis, the source metadata characterizing the input data;
• metadata ingestion circuitry executable by the processor to normalize the source metadata into a universal metadata repository by reconciliation of disparate formats of the source metadata into a common schema format;
• metadata analytics circuitry executable by the processor to perform data analytics on the input data driven by the universal metadata repository, performance of the data analytics comprising:
• data quality metrics analysis and data lineage analysis of the source metadata;
• identification of key data frames based on the data quality metrics analysis and the data lineage analysis, the key data frames being a sample of larger amounts of metadata included in the source metadata;
• storage of the key data frames, in the common schema format, instead of the source metadata, in the universal metadata repository, the key data frames including attributes providing a data source of the source metadata represented by the key data frames;
• generation of an incidence graph schema to model a relational composition of metadata represented by the key data frames, the key data frames depicted in the incidence graph schema;
• and
• event manager circuitry executable by the processor to execute a feedback loop responsive to the data analytics to deliver a input data feedback message to at least one of the data sources, wherein the input data feedback message specifies a data quality alert responsive to the data quality metrics, the data quality alert being responsive to a data quality rule executed on the data quality metrics.

A non-transitory computer readable medium comprising instructions executable by a processor, the instructions comprising:

• instructions executable by the processor to select a plurality of data sources that provide input data to an identified pre-defined workflow;
• instructions executable by the processor to ingest a dataset of metadata from each of the data sources, the dataset of metadata characterizing the input data provided to the pre-defined workflow;
• instructions executable by the processor to normalize the ingested dataset of metadata to a uniform schema by reconciliation of disparate formats of the source metadata into a common schema format of a universal metadata repository;
• instructions executable by the processor to perform data quality analysis and data lineage analysis;
• instructions executable by the processor to identify a key data frame in each of the ingested datasets of metadata based on the data quality analysis and the data lineage analysis;
• instructions executable by the processor to store only the key data frame, in the common schema format, in the universal metadata repository in association with a timestamp indicating when a respective dataset of metadata was ingested and attributes providing a data source of the ingested dataset of metadata represented by the key data frame, each key data frame being identified due to being a representative sample of remaining metadata in the respective ingested dataset of metadata, the remaining metadata in the respective ingested dataset of metadata omitted from storage in the universal metadata repository;
• instructions executable by the processor to generate an incidence graph schema modeling a relational composition of metadata represented by the key data frames, the incidence graph schema comprising a technical context graph and a business context graph generated based on the universal metadata repository and including a depiction of the key data frames;
• instructions executable by the processor to generate a feedback message for transmission to one of the plurality of data sources, the feedback message comprising a data quality alert responsive to data quality metrics, the data quality alert being responsive to a data quality rule executed on the data quality metrics.