Back to results
US 11,893,029 B2Granted
Real-time streaming data ingestion into database tables
AssigneeSnowflake Inc.
InventorsTyler Arthur Akidau, Istvan Cseri, Tyler Jones, Daniel E. Sotolongo, Zhuo Zhang
Life of this patent
Estimating the term…
Abstract
A streaming ingest platform can improve latency and expense issues related to uploading data into a cloud data system. The streaming ingest platform can organize the data to be ingested into per-table chunks and per-account blobs. This data may be committed and may be made available for query processing before it is ingested into the target source tables. This significantly improves latency issues. The streaming ingest platform can also accommodate uploading data from various sources with different processing and communication capabilities, such as Internet of Things (IOT) devices.
Claims18
A method comprising:
- receiving a registration request for a per-account group of files, the registration request including identification information for the per-account group of files and a storage location of the per-account group of files;
- accessing the per-account group of files based on the registration request, the per-account group of files being stored in a first format;
- deduping and validating the per-account group of files using sequencing information included in the per-account group of files;
- committing the per-account group of files stored in the storage location and making data in the per-account group of files in the first format accessible for query processing before the data is ingested into one or more source tables;
- generating a hybrid table for query processing, the hybrid table including the committed data in the first format and data from the one or more source tables;
- receiving a query prior to the data being ingested;
- executing the query using the hybrid table to generate results for the query including:
- converting the committed data from the first format into a common format;
- converting the data from the one or more source tables into the common format;
- joining the committed data in the common format and the data from the one or more source tables in the common format to generate joined data;
- and
- executing the query based on the joined data;
- and
- ingesting the data into the one or more source tables in a second format.
11 more elements
A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
- receiving a registration request for a per-account group of files, the registration request including identification information for the per-account group of files and a storage location of the per-account group of files;
- accessing the per-account group of files based on the registration request, the per-account group of files being stored in a first format;
- deduping and validating the per-account group of files using sequencing information included in the per-account group of files;
- committing the per-account group of files stored in the storage location and making data in the per-account group of files in the first format accessible for query processing before the data is ingested into one or more source tables;
- generating a hybrid table for query processing, the hybrid table including the committed data in the first format and data from the one or more source tables;
- receiving a query prior to the data being ingested;
- executing the query using the hybrid table to generate results for the query including:
- converting the committed data from the first format into a common format;
- converting the data from the one or more source tables into the common format;
- joining the committed data in the common format and the data from the one or more source tables in the common format to generate joined data;
- and
- executing the query based on the joined data;
- and
- ingesting the data into the one or more source tables in a second format.
11 more elements
A system comprising:
- at least one hardware processor;
- and
- at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
- receiving a registration request for a per-account group of files, the registration request including identification information for the per-account group of files and a storage location of the per-account group of files;
- accessing the per-account group of files based on the registration request, the per-account group of files being stored in a first format;
- deduping and validating the per-account group of files using sequencing information included in the per-account group of files;
- committing the per-account group of files stored in the storage location and making data in the per-account group of files in the first format accessible for query processing before the data is ingested into one or more source tables;
- generating a hybrid table for query processing, the hybrid table including the committed data in the first format and data from the one or more source tables;
- receiving a query prior to the data being ingested;
- executing the query using the hybrid table to generate results for the query including:
- converting the committed data from the first format into a common format;
- converting the data from the one or more source tables into the common format;
- joining the committed data in the common format and the data from the one or more source tables in the common format to generate joined data;
- and
- executing the query based on the joined data;
- and
- ingesting the data into the one or more source tables in a second format.
14 more elements
Dependencies are inferred from the claim text.