Cloud platform based data mesh architecture for data pipelines

A non-transitory computer readable storage medium storing instructions that when executed by a computer processor, cause the computer processor to perform steps comprising:

• receiving a specification of a data pipeline configured as a data mesh representing a directed acyclic graph of a plurality of nodes connected by edges, wherein each node of the data mesh stores snapshots of partially computed data in one or more storage units of the respective node, each snapshot associated with a respective time value;
• for each node of the data mesh, generating initial instructions for configuring the node on a respective one of one or more identified cloud platforms based on the specification, wherein the specification is cloud platform independent, and wherein the initial instructions are specific to the identified cloud platform;
• deploying the data pipeline at each of the one or more identified cloud platforms;
• responsive to receiving input data for the data mesh, executing the initial instructions corresponding to the plurality of nodes, wherein the data generated by each node is propagated according to the connection;
• generating lineage information describing a field, wherein the field is one of:
• an input field, wherein the lineage information represents a set of nodes determining values derived from the input field;
• and
• an output field, wherein the lineage information represents a set of nodes determining values used to compute the output field;
• receiving a change to the specification describing the field of change;
• identifying a subset of nodes of the data mesh based on the lineage information of the field affected by the change;
• recommending the subset of nodes of the data mesh as data pipeline units that need to be modified in connection with the change;
• receiving a modified specification for the recommended set of nodes of the data mesh;
• generating modified instructions for the recommended set of nodes the data mesh;
• for only the recommended set of nodes, executing the modified instructions to reconfigure the recommended set of nodes according to the modified specification such that a remainder of the nodes of the data mesh remain unmodified;
• receiving a second time value and a same or different subset of nodes of the data mesh;
• and
• for only each of the same or different subset of nodes of the data mesh, executing the modified instructions of the node to re-process the partially computed data stored on the node obtained from the snapshot associated with the time value corresponding to the second time value.

A computing system comprising:

• a computer processor;
• and
• a non-transitory computer readable storage medium storing instructions that when executed by a computer processor, cause the computer processor to perform steps of a computer-implemented method for a continuous delivery of a data pipeline on a cloud platform, the steps comprising:
• receiving a specification of a data pipeline configured as a data mesh representing an acyclic graph of a plurality of connected nodes, wherein each node of the data mesh stores partially computed data snapshots in one or more storage units of the respective node associated with a respective first time value, thereby providing persisted historical snapshots of the partially computed data;
• for each node of the data mesh, generating initial instructions for configuring the node on a respective one of one or more identified cloud platform s based on the specification, wherein the specification is cloud platform independent, and wherein the initial instructions are specific to the identified cloud platform;
• deploying the data pipeline at each of the one or more identified cloud platforms;
• responsive to receiving input data for the data mesh, executing the initial instructions corresponding to the plurality of nodes, wherein the data generated by each node is propagated according to the connection;
• generating information describing a field, wherein the field is at least one of:
• an input field, wherein the information represents a set of nodes determining values derived from the input field;
• and
• an output field, wherein the information represents a set of nodes determining values used to compute the output field;
• receiving a change to the specification;
• identifying a subset of nodes of the data mesh based on the information of the field affected by the change;
• identifying the subset of nodes as data pipeline units that need to be modified in connection with the change;
• receiving a modified specification for the subset of nodes;
• generating modified instructions for the subset of nodes;
• for the subset of nodes, executing the modified instructions to reconfigure the subset of nodes according to the modified specification such that a remainder of the nodes of the data mesh remain unmodified;
• receiving a second time value and a same or different subset of nodes of the data mesh;
• and
• for each of the same or different subset of nodes, executing the modified instructions to re-process the persisted historical snapshots of the partially computed data associated with the second time value.