Efficient multi-part upload for a data warehouse
AssigneeAmazon Technologies, Inc.
InventorChris Alan Keyser
Life of this patent
Estimating the term…
Abstract
Data may be partitioned and uploaded in multiple parts in parallel to a data warehouse cluster in a data warehouse system. Data to be uploaded may be identified, and the partitions for the data may be determined at the storage client. The data may then be partitioned at the storage client. In various embodiments, no local partitions of the data may be maintained in persistent storage at the storage client. The partitioned data may then be sent in parallel to a data warehouse staging area in another network-based service that is implemented as part of a same network-based service implementing the data warehouse system. A request may then be sent to the data warehouse cluster to perform a multi-part upload from the staging area to the data warehouse cluster.
Claims20
A system, comprising:
- a plurality of compute nodes implementing a network-based services platform;
- at least some compute nodes of the plurality of compute nodes configured to implement a data warehouse cluster as part of a data warehouse service provided by the network-based services platform, wherein the data warehouse cluster provides data storage among the at least some compute nodes according to a data distribution scheme;
- another one or more compute nodes of the plurality of compute nodes configured to implement an upload staging area for the data warehouse cluster, wherein the upload staging area is accessible to the data warehouse cluster as part of the network-based services platform;
- at least one other compute node of the plurality of compute nodes configured to provide a dynamic, multi-part upload module from the network-based services platform to a storage client of the data warehouse cluster;
- the dynamic, multi-part upload module, configured to:
- determine, at the storage client, a plurality of partitions for data maintained at the storage client to be uploaded to the data warehouse cluster according to the data distribution scheme for the at least some compute nodes in the data warehouse cluster;
- dynamically partition the data at the storage client according to the determined plurality of partitions;
- send the dynamically partitioned data from the storage client to the upload staging area for the data warehouse cluster;
- and
- subsequent to said sending the partitioned data, send, from the storage client, an upload request to the data warehouse cluster in order to upload the plurality of partitions of the data from the upload staging area to respective ones of the at least some compute nodes in the data warehouse cluster;
- at least some compute nodes of the plurality of compute nodes of the network-based services platform, in response to receipt of the upload request, upload respective partitions of the plurality of partitions of the data in parallel from the upload staging area to respective ones of the at least some compute nodes in the data warehouse cluster.
A method, comprising:
- performing, by one or more computing devices implementing a storage client:
- identifying, at the storage client, data to be uploaded to a data warehouse cluster from the storage client, wherein the data warehouse cluster provides data storage among a plurality of compute nodes according to a data distribution scheme;
- determining, at the storage client, a plurality of partitions for the data according to the data distribution scheme for the plurality of compute nodes in the data warehouse cluster;
- partitioning, at the storage client, the data according to the determined plurality of partitions;
- sending, from the storage client, the partitioned data in parallel to an upload staging area that is accessible to the plurality of compute nodes of the data warehouse cluster;
- and
- subsequent to said sending the partitioned data, sending, from the storage client, an upload request to the data warehouse cluster for a multi-part upload of the partitioned data from the upload staging area to respective ones of the plurality of compute nodes in the data warehouse cluster.
A non-transitory, computer-readable storage medium, comprising program instructions that implement a client-side, dynamic, multi-part upload module and that when executed by one or more computing devices cause the one or more computing devices to implement:
- identifying, at a storage client, data to be uploaded to a data warehouse cluster from the storage client, wherein the data warehouse cluster provides data storage among a plurality of compute nodes according to a data distribution scheme;
- determining, at the storage client, a plurality of partitions for the data according to the data distribution scheme for the plurality of compute nodes in the data warehouse cluster;
- partitioning, at the storage client, the data according to the determined plurality of partitions;
- sending, from the storage client, the partitioned data in parallel to an upload staging area that is accessible to the plurality of compute nodes of the data warehouse cluster;
- and
- subsequent to said sending the partitioned data, sending, from the storage client, an upload request to the data warehouse cluster for a multi-part upload of the partitioned data from the upload staging area to respective ones of the plurality of compute nodes in the data warehouse cluster.
Dependencies are inferred from the claim text.