Database processing using hybrid key-value tables
AssigneeSnowflake Inc.
InventorsJoshua Slocum, Evan J. Tschannen
Life of this patent
Estimating the term…
Abstract
A distributed database system can include transactional database and an object storage database. The data of the transactional database can be split into granules and replicated to the object storage database. The distributed database system can process transactional requests using the transactional database. The distributed database can receive a request that reads data more than a set size from the transactional database. The distributed database system can identify the granule data in the object storage database and transmit data to complete the read on one or more of a plurality of execution nodes.
Claims24
A method for processing data on a distributed database:
- storing, by one or more hardware processors, key-value data in a transactional database of the distributed database, the key-value data being stored in a key-value pair format in the transactional database;
- querying the transactional database to determine a range size of the key-value data;
- splitting, on the transactional database, the key-value data into range granules that cover data ranges, the range granules being indexed by range based on the range size, at least a portion of the range granules being of equal size;
- replicating the range granules to a range-based object storage database of the distributed database, the replicating of the range granules including storing the range granules in an object format in the range-based object storage database;
- receiving a read request on the transactional database;
- determining that the read request is a read of the key-value data that exceeds a size limit;
- in response to determining that the read request exceeds the size limit for the key-value data, identifying the range granules in the range-based object storage database that correspond to data requested in the read request, and transmitting a list of object data of the identified range granules to a plurality of execution nodes to process the read request;
- receiving, from the range-based object storage database, the range granules that correspond to the list of object data using the plurality of execution nodes;
- and
- generating, by the plurality of execution nodes, results data according to the read request using the range granules received from the range-based object storage database.
A system comprising:
- one or more processors of a machine;
- and
- memory-storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:
- storing key-value data in a transactional database of a distributed database, the key-value data being stored in a key-value pair format in the transactional database;
- querying the transactional database to determine a range size of the key-value data;
- splitting, on the transactional database, the key-value data into range granules that cover data ranges, the range granules being indexed by range based on the range size, at least a portion of the range granules being of equal size;
- replicating the range granules to a range-based object storage database of the distributed database, the replicating of the range granules including storing the range granules in an object format in the range-based object storage database;
- receiving a read request on the transactional database;
- determining that the read request is a read of the key-value data that exceeds a size limit;
- in response to determining that the read request exceeds the size limit for the key-value data, identifying the range granules in the range-based object storage database that correspond to data requested in the read request, and transmitting a list of object data of the identified range granules to a plurality of execution nodes to process the read request;
- receiving, from the range-based object storage database, the range granules that correspond to the list of object data using the plurality of execution nodes;
- and
- generating, by the plurality of execution nodes, results data according to the read request using the range granules received from the range-based object storage database.
A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
- storing key-value data in a transactional database of a distributed database, the key-value data being stored in a key-value pair format in the transactional database;
- querying the transactional database to determine a range size of the key-value data;
- splitting, on the transactional database, the key-value data into range granules that cover data ranges, the range granules being indexed by range based on the range size, at least a portion of the range granules being of equal size;
- replicating the range granules to a range-based object storage database of the distributed database, the replicating of the range granules including storing the range granules being stored in an object format in the range-based object storage database;
- receiving a read request on the transactional database;
- determining that the read request is a read of the key-value data that exceeds a size limit;
- in response to determining that the read request exceeding exceeds the size limit for the key-value data, identifying the range granules in the range-based object storage database that correspond to data requested in the read request, and transmitting a list of object data of the identified range granules to a plurality of execution nodes to process the read request;
- receiving, from the range-based object storage database, the range granules that correspond to the list of object data using the plurality of execution nodes;
- and
- generating, by the plurality of execution nodes, results data according to the read request using the range granules received from the range-based object storage database.
Dependencies are inferred from the claim text.