System and method for data event management system

A system comprising:

• a data portal that receives a data event;
• wherein the data event is an event associated with data that is located in one or more data stores from among a plurality of heterogeneous data stores of an enterprise;
• and
• wherein the data portal is user application accessible interface with which users can interact to access the heterogeneous data stores, and to access information about the heterogeneous data stores without accessing underlying data of the heterogeneous data stores;
• an event pipeline service that conducts additional processing on the data event, wherein the additional processing includes:
• logging the data event via a pipeline service in a data event log;
• managing execution, via the pipeline service, of a plurality of analyses associated with the data event, the plurality of analyses each generating at least one of usage or timeliness data for the data event, wherein the timeliness data includes tracking time related statistics as the data flows through the data enterprise;
• and
• transmitting a result of the analyses to at least one application external to the data portal, the result including a trend analysis of data events and a recommendation regarding restructuring of a query syntax to comply with a predetermined query syntax standard and wherein transmitting includes, at one or more event distributors, enabling using one or more applications to listen to the data event processed at the data portal during the processing;
• an event ingest service that ingests the data event into the event pipeline service for the additional processing.

A method comprising:

• receiving a data event associated with a read action query at a data portal,
• wherein the data event is an event associated with data that is located in one or more data stores from among a plurality of heterogeneous data stores of a data ecosystem;
• and
• wherein the data portal is a user application accessible interface with which users can interact to access the heterogeneous data stores, and to access information about the heterogeneous data stores without accessing underlying data of the heterogeneous data stores;
• processing the data event at the data portal;
• ingesting the data event into a pipeline service for additional processing;
• logging the data event via a pipeline service in a data event log;
• managing execution, via the pipeline service, of a plurality of analyses associated with the data event, wherein:
• the plurality of analyses are each implemented using an independent micro-services architecture;
• and
• the plurality of analyses includes:
• generation of at least one of usage or timeliness data for the data event, wherein the timeliness data includes tracking time related statistics as the data flows through the data ecosystem;
• and
• a common data join analysis that comprises:
• parsing the read action query;
• analyzing data stores associated with the read action query for usage data;
• and
• based on the analysis, generating a recommendation to restructure the read action query syntax to comply with a predetermined query syntax standard and join one or more data sets distributed across one or more data stores to improve performance of the read action query;
• and
• transmitting the recommendation to at least one application external to the data portal, wherein transmitting includes, at one or more event distributors, enabling one or more applications to listen to the data event processed at the data portal during the processing.