Systems and methods for improved blockchain data indexing by decoupling compute and storage layers
AssigneeCoinbase, Inc.
InventorsJie ZHANG, Zhicong LIANG, Yaohua YANG, David LAI, Chaoqing LU, Jinghan XU, Xu MENG
Life of this patent
Estimating the term…
Abstract
Systems and methods for a unified approach that is compatible with all blockchains, protocols, etc. The systems and methods use a bifurcated indexing system with a dynamically selected application service. The systems and methods bifurcate the indexing process into a storage layer and a compute layer.
Claims20
A system for improved blockchain data indexing by decoupling compute and storage layers, the system comprising:
- a multi-layer data platform for indexing on-chain data;
- a first dataset, wherein the first dataset is populated based on:
- receiving first on-chain data from a blockchain node of a blockchain network, wherein the first on-chain data comprises hexadecimal encoded data from a first block of the blockchain network;
- transforming the first on-chain data to a first format, and wherein the first format comprises data types with field names identified by a respective integer;
- storing the first on-chain data, in the first dataset at a blockchain-interface layer, wherein the first dataset comprises the hexadecimal encoded data, and wherein the first dataset uses the first format;
- retrieving a first compute engine, wherein the first compute engine comprises a first workflow architecture, wherein the first workflow architecture comprises a first threshold for workflow throughout and a first threshold for a number of workflows;
- a second dataset, wherein the second dataset is populated based on:
- transforming the first on-chain data in the first format to a second format, wherein the second format comprises a columnar oriented format;
- storing the first on-chain data, in the second dataset at a data lakehouse layer, wherein the second dataset comprises the first on-chain data and second on-chain data in the second format, and wherein the second on-chain data is from a second block on the blockchain network;
- retrieving a second compute engine, wherein the second compute engine comprises a second workflow architecture, wherein the second workflow architecture comprises a second threshold for workflow throughout and a second threshold for the number of workflows, wherein the second threshold for workflow throughput is higher than the first threshold for workflow throughput, and wherein the second threshold for the number of workflows is lower than the first threshold for the number of workflows;
- and
- a third dataset, wherein the third dataset is populated based on:
- transforming the first on-chain data and the second on-chain data to a third format, wherein the third format is dynamically selected based on an application;
- storing the first on-chain data and the second on-chain data, in the third dataset at an application service layer;
- retrieving a third compute engine, wherein the third compute engine comprises the second workflow architecture.
A method for improved blockchain data indexing by decoupling compute and storage layers, the method comprising:
- receiving, at a blockchain-interface layer, first on-chain data from a blockchain node of a blockchain network, wherein the first on-chain data comprises hexadecimal encoded data from a first block of the blockchain network, wherein the blockchain-interface layer transforms the first on-chain data to a first format, using a first compute engine, for storage in a first dataset, and wherein the first format comprises data types with field names identified by a respective integer;
- receiving, at a data lakehouse layer, the first on-chain data in the first format, wherein the data lakehouse layer transforms the first on-chain data to a second format, using a second compute engine, for storage in a second dataset, wherein the second format comprises a columnar oriented format, wherein the second dataset comprises the first on-chain data and second on-chain data, and wherein the second on-chain data is from a second block on the blockchain network;
- determining an application characteristic for an application that performs blockchain operations using the first on-chain data or the second on-chain data;
- receiving, at an application service layer, the first on-chain data and the second on-chain data in the second format, wherein the application service layer transforms, using a third compute engine, the first on-chain data and the second on-chain data to a third format for storage in a third dataset, and wherein the third format is dynamically selected based on the application characteristic;
- and
- transmitting the first on-chain data and the second on-chain data in the third format to the application.
A non-transitory computer-readable medium having instructions recorded thereon that when executed by one or more processors causes operations comprising:
- receiving, at a blockchain-interface layer, first on-chain data from a blockchain node of a blockchain network, wherein the first on-chain data comprises hexadecimal encoded data from a first block of the blockchain network, wherein the blockchain-interface layer transforms the first on-chain data to a first format, using a first compute engine, for storage in a first dataset, and wherein the first format comprises data types with field names identified by a respective integer;
- receiving, at a data lakehouse layer, the first on-chain data in the first format, wherein the data lakehouse layer transforms the first on-chain data to a second format, using a second compute engine, for storage in a second dataset, wherein the second format comprises a columnar oriented format, wherein the second dataset comprises the first on-chain data and second on-chain data, and wherein the second on-chain data is from a second block on the blockchain network;
- determining an application characteristic for an application that performs blockchain operations using the first on-chain data or the second on-chain data;
- receiving, at an application service layer, the first on-chain data and the second on-chain data in the second format, wherein the application service layer transforms, using a third compute engine, the first on-chain data and the second on-chain data to a third format for storage in a third dataset, and wherein the third format is dynamically selected based on the application characteristic;
- and
- transmitting the first on-chain data and the second on-chain data in the third format to the application.
Dependencies are inferred from the claim text.