Back to results
US 2021/0279209 A1Published application
Readable and writable streams on external data sources
AssigneeSnowflake Inc.
InventorsIstvan Cseri, Tyler Jones
Life of this patent
Estimating the term…
Abstract
The subject technology receives an operation to perform on an external data source accessible via a network, the external data source being hosted by an external system separate from a network-based database system. The subject technology determines a set of shards corresponding to the external data source. The subject technology determines a set of offsets of each shard of the set of shards. The subject technology, based on the set of shards and the set of offsets, performs the operation on the external data source. The subject technology provides an indication that the operation is complete.
Claims30
A network-based database system comprising:
- at least one hardware processor;
- and
- a memory storing instructions that cause the at least one hardware processor to perform operations comprising:
- receiving an operation to perform on an external data source accessible via a network, the external data source being hosted by an external system separate from the network-based database system, the operation comprising a query statement on the external data source, the query statement comprising a write operation;
- initializing a connection to the external data source using an external integration to establish the connection to the external data source, the external integration including security and configuration information, the security and configuration information comprising a retry timeout;
- determining, using the connection to the external data source, a set of shards stored in the external data source;
- determining a set of offsets of each shard of the set of shards;
- based on the set of shards and the set of offsets, performing the operation on the external data source by performing, using the connection to the external data source established via the external integration, the write operation from the query statement on the external data source, the external data source being different than a storage platform associated with the network-based database system;
- receiving an indication from the external data source that the write operation was successful;
- and
- storing a second indication that the write operation is completed on the storage platform associated with the network-based database system.
8 more elements
(canceled)
A method comprising:
- receiving, using at least one hardware processor, an operation to perform on an external data source accessible via a network, the external data source being hosted by an external system separate from a network-based database system, the operation comprising a query statement on the external data source, the query statement comprising a write operation;
- initializing, using the at least one hardware processor, a connection to the external data source using an external integration to establish the connection to the external data source, the external integration including security and configuration information, the security and configuration information comprising a retry timeout;
- determining, using the at least one hardware processor and the connection to the external data source, a set of shards stored in the external data source;
- determining, using the at least one hardware processor, a set of offsets of each shard of the set of shards;
- based on the set of shards and the set of offsets, performing, using the connection to the external data source established via the external integration, the operation on the external data source by performing the write operation from the query statement on the external data source, the external data source being different than a storage platform associated with the network-based database system;
- receiving an indication from the external data source that the write operation was successful;
- and
- storing a second indication that the write operation is completed on the storage platform associated with the network-based database system.
5 more elements
(canceled)
A non-transitory computer-storage medium comprising instructions that, when executed by a processing device, configure the processing device to perform operations comprising:
- receiving an operation to perform on an external data source accessible via a network, the external data source being hosted by an external system separate from a network-based database system, the operation comprising a query statement on the external data source, the query statement comprising a write operation;
- initializing a connection to the external data source using an external integration to establish the connection to the external data source, the external integration including security and configuration information, the security and configuration information comprising a retry timeout;
- determining, using the connection to the external data source, a set of shards stored in the external data source;
- determining a set of offsets of each shard of the set of shards;
- based on the set of shards and the set of offsets, performing the operation on the external data source by performing, using the connection to the external data source established via the external integration, the write operation from the query statement on the external data source, the external data source being different than a storage platform associated with the network-based database system;
- receiving an indication from the external data source that the write operation was successful;
- and
- storing a second indication that the write operation is completed on the storage platform associated with the network-based database system.
5 more elements
(canceled)
Dependencies are inferred from the claim text.