# US 2024/0202208 A1 · End-to-end database management framework

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0202208 A1
- Document ID: US20240202208A1
- Status: Published application
- Kind: A1
- Application number: 18/082,563
- Filed: 2022-12-15
- Published: 2024-06-20
- Earliest claimed: 2022-12-15
- Assignees: Patterns Data Systems Inc.
- Applicants: Patterns Data Systems Inc.
- Inventors: Kenneth Van Haren; Christopher Cziraky-Stanley
- Web page: https://patentgrep.com/patent/US20240202208A1

## CPC

- G06F 16/258: {Data format conversion from or to a database}
- G06F 16/254: {Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses}
- G06F 16/26: Visual data mining; Browsing structured data
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An integrated end-to-end data management programming and environment, as well as methods of end-to-end data management are described. A user of the described systems and methods can design and deploy a data pipeline project to import data from third party data sources, to perform data functions, and to store output data in a data warehouse. The user can write data pipeline projects, applications, and programs in the form of data graphs, where graph nodes perform the data pipeline functions. In some embodiments, the user can download entire task- or server-specific data graphs or portions of a data graph from a library of pre-configured objects.

## Claims

### Claim 1

A method comprising:
receiving, from a user, a plurality of nodes, the nodes comprising an end-to-end data pipeline graph, wherein each node comprises an operation of the data pipeline, the operations comprising ingesting data from a third-party server into an external data warehouse;
receiving, from the user, a graph configuration file comprising inputs and outputs of each node;
uploading the nodes and the graph configuration file to a database management (DMS) server;
detecting graph edges by parsing the nodes and/or the graph configuration file;
detecting input/output pairs between the nodes, wherein each pair of input/output comprises an edge of the graph, wherein the edges further determine execution dependencies of the nodes;
executing the nodes according to the graph edges and the graph configuration file;
generating an output in the external data warehouse.

### Claim 2 (depends on 1)

The method of claim 1, wherein the nodes comprise a third-party-specific importer node configured to interface with an application programming interface of the third-party server and import data from the third-party server to the data warehouse.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
providing a library of nodes comprising preconfigured data pipeline graphs specific to the third-party server and/or a data pipeline project related to the third-party server;
enabling the user to clone and/or modify a node, a collection of nodes, and/or a preconfigured data pipeline graph downloaded from the library.

### Claim 4 (depends on 1)

The method of claim 1, wherein executing the nodes further comprise a completed node sending a message via a message queue, the message comprising an indication of completion of an execution by the completed node, wherein upon receiving the message, the DMS server executes nodes having execution dependencies on the completed node.

### Claim 5 (depends on 1)

The method of claim 1 further comprising:
providing a user interface, comprising a graph panel, the graph panel providing a visual representation of the end-to-end data pipeline graph by:
displaying a plurality of rectangles, each rectangle representing a node of the graph,
connecting the rectangles with lines based on whether a graph edge exists between the nodes corresponding to the rectangles.

### Claim 6 (depends on 1)

The method of claim 1, wherein the nodes further comprise:
an importer node configuring the external data warehouse to perform operations comprising:
ingesting data from the third-party server;
generating a table from the imported data, based on a selected format; and
a chart node generating a chart based on the table and configuring the external data warehouse to store the chart.

### Claim 7 (depends on 1)

The method of claim 1, wherein the nodes further comprise one or more of a single record node and streaming node, wherein the single record node configures the external data warehouse to write single records into an output table in the data warehouse and the streaming node configures the external data warehouse to write multiple records in the output table in the data warehouse.

### Claim 8 (depends on 1)

The method of claim 1 further comprising:
providing a library of nodes, wherein the nodes are preconfigured to import data from a plurality of third-party servers and generate standardized data objects, based on semantic meaning of the ingested data, such that data imported from the plurality of the third-party servers is presented in a common format in the standardized data object.

### Claim 9

A non-transitory computer storage that stores executable program instructions that, when executed by one or more computing devices, configure the one or more computing devices to perform operations comprising:
receiving, from a user, a plurality of nodes, the nodes comprising an end-to-end data pipeline graph, wherein each node comprises an operation of the data pipeline, the operations comprising ingesting data from a third-party server into an external data warehouse;
receiving, from the user, a graph configuration file comprising inputs and outputs of each node;
uploading the nodes and the graph configuration file to a database management (DMS) server;
detecting graph edges by parsing the nodes and/or the graph configuration file;
detecting input/output pairs between the nodes, wherein each pair of input/output comprises an edge of the graph, wherein the edges further determine execution dependencies of the nodes;
executing the nodes according to the graph edges and the graph configuration file;
generating an output in the external data warehouse.

### Claim 10 (depends on 9)

The non-transitory computer storage of claim 9, wherein the nodes comprise a third-party-specific importer node configured to interface with an application programming interface of the third-party server and import data from the third-party server to the data warehouse.

### Claim 11 (depends on 9)

The non-transitory computer storage of claim 9, wherein the operations further comprise:
providing a library of nodes comprising preconfigured data pipeline graphs specific to the third-party server and/or a data pipeline project related to the third-party server;
enabling the user to clone and/or modify a node, a collection of nodes, and/or a preconfigured data pipeline graph downloaded from the library.

### Claim 12 (depends on 9)

The non-transitory computer-storage of claim 9, wherein executing the nodes further comprise a completed node sending a message via a message queue, the message comprising an indication of completion of an execution by the completed node, wherein upon receiving the message, the DMS server executes nodes having execution dependencies on the completed node.

### Claim 13 (depends on 9)

The non-transitory computer storage of claim 9, wherein the operations further comprise:
providing a user interface, comprising a graph panel, the graph panel providing a visual representation of the end-to-end data pipeline graph by:
displaying a plurality of rectangles, each rectangle representing a node of the graph,
connecting the rectangles with lines based on whether a graph edge exists between the nodes corresponding to the rectangles.

### Claim 14 (depends on 9)

The non-transitory computer-storage of claim 9, wherein the nodes further comprise:
an importer node configuring the external data warehouse to perform operations comprising:
ingesting data from the third-party server;
generating a table from the imported data, based on a selected format; and
a chart node generating a chart based on the table and configuring the external data warehouse to store the chart.

### Claim 15 (depends on 9)

The non-transitory computer-storage of claim 9, wherein the nodes further comprise one or more of a single record node and streaming node, wherein the single record node configures the external data warehouse to write single records into an output table in the data warehouse and the streaming node configures the external data warehouse to write multiple records in the output table in the data warehouse.

### Claim 16 (depends on 9)

The non-transitory computer storage of claim 9, wherein the operations further comprise:
providing a library of nodes, wherein the nodes are preconfigured to import data from a plurality of third-party servers and generate standardized data objects, based on semantic meaning of the ingested data, such that data imported from the plurality of the third-party servers is presented in a common format in the standardized data object.

### Claim 17

A data management server (DMS) configured to perform operations comprising:
receiving, from a user, a plurality of nodes, the nodes comprising an end-to-end data pipeline graph, wherein each node comprises an operation of the data pipeline, the operations comprising ingesting data from a third-party server into an external data warehouse;
receiving, from the user, a graph configuration file comprising inputs and outputs of each node;
detecting graph edges by parsing the nodes and/or the graph configuration file;
detecting input/output pairs between the nodes, wherein each pair of input/output comprises an edge of the graph, wherein the edges further determine execution dependencies of the nodes;
executing the nodes according to the graph edges and the graph configuration file;
generating an output in the external data warehouse.

### Claim 18 (depends on 17)

The system of claim 17, wherein the nodes comprise a third-party-specific importer node configured to interface with an application programming interface of the third-party server and import data from the third-party server to the data warehouse.

### Claim 19 (depends on 17)

The system of claim 17, further comprising:
a library of nodes comprising preconfigured data pipeline graphs specific to the third-party server and/or a data pipeline project related to the third-party server, wherein the user is enabled to clone and/or modify a node, a collection of nodes, and/or a preconfigured data pipeline graph downloaded from the library.

### Claim 20 (depends on 17)

The system of claim 17, wherein executing the nodes further comprise a completed node sending a message via a message queue, the message comprising an indication of completion of an execution by the completed node, wherein upon receiving the message, the DMS server executes nodes having execution dependencies on the completed node.
