Local storage clustering for redundancy coded data storage system
AssigneeAmazon Technologies, Inc.
InventorsColin Laird Lazier, Adam Frederick Brock
Life of this patent
Estimating the term…
Abstract
A data transfer device is used to augment the capabilities of a data storage system. The data transfer device may be capable of persistently storing data for an indeterminate amount of time, and may be configured to store a portion of a bundle of redundancy coded shards that span between the data transfer device and a data storage system configured to store the remainder of the bundle. Data stored on the data transfer device may be read from and written directly to the data transfer device without transfer of data to the data storage system. If the data transfer device is not available, the remaining shards of the bundle may provide a regenerated, original form of the data.
Claims20
A computer-implemented method, comprising:
- obtaining a data storage request sent to an interface of a plurality of data transfer devices, the interface being shared with a data storage system, the plurality of data transfer devices including a local version of the interface corresponding to a remote interface provided by the data storage system, the data storage request requesting data storage at the data storage system while an operable connection between the data storage system and a customer device associated with the data storage request is unavailable;
- generating, by the plurality of data transfer devices and based at least in part on data of the data storage request, a plurality of redundancy encoded shards such that a subset of the plurality of redundancy encoded shards, each having less than all the data, is sufficient to reconstruct the data;
- and
- distributing the plurality of redundancy encoded shards among at least a subset of the plurality of data transfer devices to be stored.
A system, comprising one or more processors and memory configured to implement one or more services, wherein the one or more services:
- provision a cluster comprising a plurality of data transfer devices, the cluster configured to provide data storage on a remote data storage service sharing a common application programming interface with the cluster, the cluster including a local version of the application programming interface corresponding to a remote application programming interface provided by the remote data storage service, the plurality of data transfer devices configured for transferring data from a customer device to the remote data storage service via an ingestion process;
- and
- cause the cluster to service a first set of requests requesting data storage on the remote data storage service, with the requests requesting data storage on the remote data storage service directed to and fulfilled by the cluster from the customer device while an operable connection between the customer device and the remote data storage service is unavailable.
A non-transitory computer-readable storage medium having stored thereon executable instructions that, if executed by one or more processors of a computer system, cause the computer system to at least:
- configure a plurality of data transfer devices to function as a cluster, the cluster providing one or more capabilities associated with a data storage system operably connected to the computer system, the cluster and the data storage system sharing a common application programming interface, the cluster including a local version of the application programming interface corresponding to an application programming interface provided by the data storage service;
- and
- cause the cluster to service requests from a requestor requesting the one or more capabilities from the data storage system, the requests fulfilled by the cluster from a customer device while an operable connection between the customer device and the data storage system is unavailable.
Dependencies are inferred from the claim text.