# US 2024/0004866 A1 · Systems and methods for supporting both batch processing and streaming data applications based on raw blockchain data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0004866 A1
- Document ID: US20240004866A1
- Status: Published application
- Kind: A1
- Application number: 17/810,091
- Filed: 2022-06-30
- Published: 2024-01-04
- Earliest claimed: 2022-06-30
- Assignees: Coinbase, Inc.
- Applicants: Coinbase, Inc.
- Inventors: Jie ZHANG; Zhicong LIANG; Yaohua YANG; David LAI; Chaoqing LU; Jinghan XU; Xu MENG
- Web page: https://patentgrep.com/patent/US20240004866A1

## CPC

- G06F 16/2379: {Updates performed during online database operations; commit processing}
- G06F 16/1794: {Details of file format conversion}
- G06F 16/258: {Data format conversion from or to a database}
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 16/273: {Asynchronous replication or reconciliation}
- G06F 16/275: {Synchronous replication}
- G06F 16/278: {Data partitioning, e.g. horizontal or vertical partitioning}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods for supporting both batch processing and streaming data applications, to load and process data incrementally, while providing a near-constantly materialized dataset based on raw blockchain data, are described. For example, the system may receive first on-chain data in a first format via a first input stream, wherein the first on-chain data originates from a blockchain node of a blockchain network. The system may transform the first on-chain data to a second format for storage in a second dataset, wherein the second format comprises an unbounded table.

## Claims

### Claim 1

A system for supporting both batch processing and streaming data applications, loaded and process data incrementally, while providing a near-constantly materialized dataset based on raw blockchain data, the system comprising:
a multi-layer data platform for indexing on-chain data, wherein the multi-layer data platform comprises an application service layer, and wherein the application service layer generates static table queries;
a first dataset, wherein the first dataset is populated based on:
receiving first on-chain data from a blockchain node of a blockchain network, wherein first on-chain data comprises hexadecimal encoded data from a first block of the blockchain network;
storing the first on-chain data, in the first dataset at a blockchain-interface layer, wherein the first dataset comprises the hexadecimal encoded data, and wherein the first dataset uses a first format;
a second dataset, wherein the second dataset is populated based on:
transforming the first on-chain data in the first format to a second format, wherein the second format comprises a columnar oriented format; and
storing the first on-chain data, in the second dataset at a data lakehouse layer, wherein the second dataset comprises the first on-chain data and second on-chain data in the second format, wherein the second on-chain data is from a second block on the blockchain network, and wherein the second format comprises:
detecting new on-chain data in the first on-chain data;
appending the new on-chain data to an unbounded table as a micro batch; and
storing the new on-chain data in the second dataset.

### Claim 2

A method for supporting both batch processing and streaming data applications, loaded and process data incrementally, while providing a near-constantly materialized dataset based on raw blockchain data, the method comprising:
receiving, at a data lakehouse layer, first on-chain data in a first format via a first input stream, wherein the first on-chain data originates from a blockchain node of a blockchain network;
transforming the first on-chain data to a second format for storage in a second dataset, wherein the second format comprises an unbounded table, and wherein transforming the first on-chain data to the second format comprises:
detecting first new on-chain data in the first input stream;
appending the first new on-chain data to the unbounded table as a micro batch; and
storing the first new on-chain data in the second dataset; and
generating an output based on the second dataset.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
receiving, via an application service layer, a static table query; and
processing the static table query using the second dataset.

### Claim 4 (depends on 2)

The method of claim 2, wherein transforming the first on-chain data to the second format further comprises:
detecting second new on-chain data in the first input stream;
modifying existing on-chain data in the unbounded table based on the second new on-chain data; and
storing the second new on-chain data in the second dataset.

### Claim 6 (depends on 2)

The method of claim 2, wherein transforming the first on-chain data to the second format further comprises:
detecting third new on-chain data in the first input stream;
modifying existing on-chain data in the unbounded table based on the third new on-chain data using a single call to insert or update the existing on-chain data in the unbounded table; and
storing the third new on-chain data in the second dataset.

### Claim 7 (depends on 2)

The method of claim 2, wherein transforming the first on-chain data to the second format further comprises:
detecting a blockchain network reorganization; and
modifying existing on-chain data in the unbounded table based on the blockchain network reorganization.

### Claim 8 (depends on 2)

The method of claim 2, wherein transforming the first on-chain data to the second format further comprises:
accessing an application binary interface for a smart contract corresponding to the first on-chain data;
determining an on-chain event in the first on-chain data based on the application binary interface; and
storing the on-chain event in the second dataset.

### Claim 9 (depends on 2)

The method of claim 2, further comprising:
receiving, at a blockchain-interface layer, the first on-chain data from the blockchain node of the blockchain network, wherein the first on-chain data comprises hexadecimal encoded data from a first block of the blockchain network; and
transforming the first on-chain data to the first format, using a first compute engine, for storage in a first dataset, wherein the first format comprises data types with field names identified by a respective integer, wherein the first compute engine comprises a first workflow architecture, and wherein the first workflow architecture comprises a first threshold for workflow throughout and a first threshold for a number of workflows.

### Claim 10 (depends on 2)

The method of claim 2, wherein the second dataset comprises the first on-chain data and second on-chain data, and wherein the second on-chain data is from a second block on the blockchain network.

### Claim 11 (depends on 2)

The method of claim 2, wherein the transforming the first on-chain data to the second format for storage in the second dataset comprises using a second compute engine, wherein the second compute engine comprises a second workflow architecture, wherein the second workflow architecture comprises a second threshold for workflow throughout and a second threshold for a number of workflows, wherein the second threshold for workflow throughput is higher than the first threshold for workflow throughput, and wherein the second threshold for the number of workflows is lower than the first threshold for the number of workflows.

### Claim 12 (depends on 2)

The method of claim 2, wherein generating the output based on the second dataset comprises generating an append-only delta table comprising added and removed events.

### Claim 13 (depends on 2)

The method of claim 2, wherein the second format comprises a columnar oriented format, and wherein appending the first new on-chain data to the unbounded table as the micro batch comprises adding a new column to the unbounded table.

### Claim 14

A non-transitory computer-readable medium having instructions recorded thereon that when executed by one or more processors causes operations comprising:
receiving, at a data lakehouse layer, first on-chain data in a first format via a first input stream, wherein the first on-chain data originates from a blockchain node of a blockchain network;
transforming the first on-chain data to a second format for storage in a second dataset, wherein the second format comprises an unbounded table, and wherein transforming the first on-chain data to the second format comprises:
detecting first new on-chain data in the first input stream;
appending the first new on-chain data to the unbounded table as a micro batch; and
storing the first new on-chain data in the second dataset; and
generating an output based on the second dataset.

### Claim 14 (depends on 14)

The non-transitory, computer-readable medium of claim 14, wherein transforming the first on-chain data to the second format further comprises:
detecting second new on-chain data in the first input stream;
modifying existing on-chain data in the unbounded table based on the second new on-chain data; and
storing the second new on-chain data in the second dataset.

### Claim 15 (depends on 14)

The non-transitory, computer-readable medium of claim 14, wherein the instructions further cause operation comprising:
receiving, via an application service layer, a static table query; and
processing the static table query using the second dataset.

### Claim 16 (depends on 14)

The non-transitory, computer-readable medium of claim 14, wherein transforming the first on-chain data to the second format further comprises:
detecting third new on-chain data in the first input stream;
modifying existing on-chain data in the unbounded table based on the third new on-chain data using a single call to insert or update the existing on-chain data in the unbounded table; and
storing the third new on-chain data in the second dataset.

### Claim 17 (depends on 14)

The non-transitory, computer-readable medium of claim 14, wherein transforming the first on-chain data to the second format further comprises:
detecting a blockchain network reorganization; and
modifying existing on-chain data in the unbounded table based on the blockchain network reorganization.

### Claim 18 (depends on 14)

The non-transitory, computer-readable medium of claim 14, wherein transforming the first on-chain data to the second format further comprises:
accessing an application binary interface for a smart contract corresponding to the first on-chain data;
determining an on-chain event in the first on-chain data based on the application binary interface; and
storing the on-chain event in the second dataset.

### Claim 19 (depends on 14)

The non-transitory, computer-readable medium of claim 14, wherein the instructions further cause operation comprising:
receiving, at a blockchain-interface layer, the first on-chain data from the blockchain node of the blockchain network, wherein the first on-chain data comprises hexadecimal encoded data from a first block of the blockchain network; and
transforming the first on-chain data to the first format, using a first compute engine, for storage in a first dataset, wherein the first format comprises data types with field names identified by a respective integer, wherein the first compute engine comprises a first workflow architecture, and wherein the first workflow architecture comprises a first threshold for workflow throughout and a first threshold for a number of workflows.

### Claim 20 (depends on 14)

The non-transitory, computer-readable medium of claim 14, wherein the second dataset comprises the first on-chain data and second on-chain data, and wherein the second on-chain data is from a second block on the blockchain network.
