Trihybrid data movement, data governance and data provenance system

A trihybrid data movement, data governance and data provenance system comprising:

• a distribute module, said distribute module comprising:
• a user interface;
• a distribute application programming interface, said distribute application programming interface communicates with the user interface;
• a distribute module database, said distribute module communicates with the distribute application programming interface, said distribute module database comprising:
• a metadata store;
• a data engine, said data engine comprising:
• a data engine application programming interface;
• a publisher flow application, said publisher flow application communicates with the data engine application programming interface;
• and
• a subscriber flow application, said subscriber flow application communicates with the publisher flow application;
• a publisher comprising:
• a job scheduler, said job scheduler publishes a publication;
• and
• a publisher datastore, said publisher datastore comprising the publication;
• a plurality of subscribers, each subscriber, included in the plurality of subscribers, comprising:
• a subscriber server, said subscriber server receives a predetermined subset of the publication, said predetermined subset specified by the subscriber;
• wherein:
• the job scheduler triggers the publishing of the publication by calling an endpoint on the distribute application programming interface, the calling comprises transmitting a unique identifier generated for the publication from the job scheduler to the distribute application programming interface;
• the distribute application programming interface retrieves from the metadata store included in the distribute module database:
• publisher metadata relating to publication, said publisher metadata comprising location of publication;
• and
• subscriber metadata, said subscriber metadata comprising:
• the plurality of subscribers;
• and
• for each of the subscribers included in the plurality of subscribers, metadata relating to the predetermined subset of the publication;
• and
• the distribute application programming interface combines the publisher metadata and the subscriber metadata into one metadata payload.

A method for trihybrid data movement, data governance and data provenance, the method comprising:

• receiving, from a publisher, at a user interface, a request to register a publication, wherein a distribute application programming interface commands the user interface;
• displaying, at the user interface, a plurality of database and/or connection questions;
• receiving, at the user interface, a set of responses to the plurality of database and/or connection questions;
• fetching, from a datastore linked to the publisher, a list of tables, columns and rows, included in the publication;
• presenting, via the user interface, the list of tables, columns and rows to the publisher;
• receiving a selection, via the user interface, of a list of tables, columns and rows to be published;
• receiving a registration confirmation from the publisher via the user interface;
• upon receipt of the registration confirmation from the publisher, transmitting a set of publication metadata to a metadata store within a distribute database via a distribute application programming interface, said set of publication metadata comprising the set of responses and the selection of the list of tables, columns and rows;
• receiving, from a subscriber, at the user interface, a request to subscribe to the publication;
• displaying to the subscriber, at the user interface, the set of publication metadata;
• receiving, from the subscriber, at the user interface, a selection of one or more tables, columns and/or rows from the set of publication metadata;
• receiving a subscription confirmation from the subscriber via the user interface;
• upon receipt of the registration confirmation from the subscriber, transmitting a set of subscription metadata to the metadata store via the distribute application programming interface, said set of subscription metadata comprising a subscriber datastore location and the selection, received from the subscriber;
• triggering, by a job scheduler, the publishing of the publication by calling an endpoint on the distribute application programming interface;
• retrieving, by the distribute application programming interface, from the metadata store, the set of publication metadata and the set of subscription metadata;
• combining the set of publication metadata and the set of subscription metadata into a metadata payload;
• transferring the metadata payload from the distribute application programming interface to a data engine application programming interface;
• identifying the publisher based on the metadata payload at the data engine application programming interface;
• calling a publication flow application from the data engine application programming interface, said calling including transferring the metadata payload to the publication flow application;
• using the metadata payload at the publication flow application to retrieve the publication from the datastore linked to the publisher;
• transmitting the publication and the set of subscription metadata from the publication flow application to a subscription flow application;
• customizing the publication into a subscriber-customized publication at the subscription flow application, said customizing based on the set of subscription metadata;
• and
• transmitting the subscriber-customized publication from the subscription flow application to a server associated with the subscriber.