System and method for real-time data dependency management and task orchestration

A system for real-time data dependency management, the system comprising:

• a processor;
• and
• a memory operatively connected to the processor via a communication interface, the memory storing computer readable instructions, when executed, causes the processor to:
• receive a plurality of data sets from a plurality of upstream applications;
• extract data entity events from the plurality of data sets;
• identify dependent data entities for each data entity event based on initializing a data dependency graph with parent data nodes that represent all entities and their respective child data dependencies;
• publish a data dependency event to an inbound topic of a data dependency service platform for each required parent data node in the data dependency graph;
• publish a data dependency ready event for a certain parent node based on determining that the certain parent node is configured for event publishing and that the certain parent node's child data dependencies are available;
• transmit the data dependency ready event to a task orchestration service platform;
• and
• orchestrate, upon receiving the data dependency ready event by the task orchestration service platform, a process instance and execute tasks for a corresponding data entity based on the process instance.

A non-transitory computer readable medium configured to store instructions for real-time data dependency management, wherein, when executed, the instructions cause a processor to perform the following:

• receiving a plurality of data sets from a plurality of upstream applications;
• extracting data entity events from the plurality of data sets;
• identifying dependent data entities for each data entity event based on initializing a data dependency graph with parent data nodes that represent all entities and their respective child data dependencies;
• publishing a data dependency event to an inbound topic of a data dependency service platform for each required parent data node in the data dependency graph;
• publishing a data dependency ready event for a certain parent node based on determining that the certain parent node is configured for event publishing and that the certain parent node's child data dependencies are available;
• transmitting the data dependency ready event to a task orchestration service platform;
• and
• orchestrating, upon receiving the data dependency ready event by the task orchestration service platform, a process instance and executing tasks for a corresponding data entity based on the process instance.