Low latency ingestion into a data system
AssigneeSnowflake Inc.
InventorsDhiraj Gupta, Subramanian Muralidhar
Life of this patent
Estimating the term…
Abstract
Described herein are techniques for improving transfer of metadata from a metadata database to a database stored in a data system, such as a data warehouse. The metadata may be written into the metadata database with a version stamp, which is monotonic increasing register value, and a partition identifier, which can be generated using attribute values of the metadata. A plurality of readers can scan the metadata database based on version stamp and partition identifier values to export the metadata to a cloud storage location. From the cloud storage location, the exported data can be auto ingested into the database, which includes a journal and snapshot table.
Claims27
A method comprising:
- receiving data into a metadata database;
- generating a version stamp for the received data;
- generating a partition identifier for the received data, the partition identifier being based on one or more attribute values of the received data;
- writing, by one or more writer computing resources, the received data into the metadata database with the version stamp and partition identifier;
- scanning, by a plurality of reader computing resources, the metadata database;
- exporting the received data to a storage location based on the version stamp and partition identifier, the plurality of readers being assigned based on partition identifiers;
- ingesting the exported data into a journal table in the database;
- and
- providing the exported data in the journal table accessible for query processing before the exported data is merged with a snapshot table.
A machine-storage medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising:
- receiving data into a metadata database;
- generating a version stamp for the received data;
- generating a partition identifier for the received data, the partition identifier being based on one or more attribute values of the received data;
- writing, by one or more writer computing resources, the received data into the metadata database with the version stamp and partition identifier;
- scanning, by a plurality of reader computing resources, the metadata database;
- exporting the received data to a storage location based on the version stamp and partition identifier, the plurality of readers being assigned based on partition identifiers;
- ingesting the exported data into a journal table in the database;
- and
- providing the exported data in the journal table accessible for query processing before the exported data is merged with a snapshot table.
A system comprising:
- at least one hardware processor;
- and
- at least one memory storing instructions that, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations comprising:
- receiving data into a metadata database;
- generating a version stamp for the received data;
- generating a partition identifier for the received data, the partition identifier being based on one or more attribute values of the received data;
- writing, by one or more writer computing resources, the received data into the metadata database with the version stamp and partition identifier;
- scanning, by a plurality of reader computing resources, the metadata database;
- exporting the received data to a storage location based on the version stamp and partition identifier, the plurality of readers being assigned based on partition identifiers;
- ingesting the exported data into a journal table in the database;
- and
- providing the exported data in the journal table accessible for query processing before the exported data is merged with a snapshot table.
Dependencies are inferred from the claim text.