Back to results
US 11,343,314 B1Granted
Stream-based logging for distributed storage systems
AssigneeAmazon Technologies, Inc.
InventorsKiran-Kumar Muniswamy-Reddy, Danny Wei
Life of this patent
Estimating the term…
Abstract
Generally described, aspects of the present application correspond to maintaining a message stream for a network-based data store, which stream includes messages reflecting modifications to the data store. Messages within the stream may be used to revert a state of the data store to a prior point in time reflected within the messages of the stream, such as by “rewinding” operations on the data store by use of the messages within the stream. Messages in the stream may further be used to asynchronously update a replica of the data store.
Claims16
A system comprising:
- a physical data store storing a representation of a volume at a first point in time;
- one or more computing devices facilitating modification to the volume and configured with computer-executable instructions to:
- receive a set of write operations to be applied to the volume;
- synchronously apply the set of write operations to a primary replica of the volume;
- generate a set of messages to be included within a stream of unbundled messages comprising write operations synchronously applied to the primary replica of the volume, but not yet applied to a secondary replica of the volume;
- and
- submit the set of messages to a logging service facilitating the stream of unbundled messages;
- and
- one or more computing devices implementing the logging service and configured with computer-executable instructions to receive the set of messages, store the set of messages within the stream of unbundled messages, cause asynchronous application of the stream of unbundled messages to the secondary replica of the volume, and dequeue messages from the stream of unbundled messages after application to the secondary replica of the volume;
- wherein the one or more computing devices facilitating modification to the volume are further configured with the computer-executable instructions to:
- receive a request to recreate a state of the volume at a second point in time subsequent to the first point in time, wherein the request comprises identification of the volume, the first point in time, and a target volume in which to recreate the state of the volume;
- store the representation of the volume at the first point in time in the target volume;
- retrieve, from the one or more computing devices implementing the logging server, a subset of messages, of the set of messages within the stream of unbundled messages, that reflect modifications to the volume between the first point in time and the second point in time;
- and
- modify the representation of the volume in the target volume by sequentially applying, to the representation of the volume, write operations reflected within the subset of messages.
12 more elements
A computer-implemented method comprising:
- receiving write operations;
- synchronously applying the write operations to a primary replica of a data volume;
- providing the write operations to a stream of messages for asynchronous application to a secondary replica of the data volume, wherein the stream of messages includes a plurality of messages and comprises a change log of sequential updates for the secondary replica, and wherein the write operations are dequeued from the stream of messages after application to the secondary replica;
- receiving a request to recreate a state of the data volume at a past point in time, wherein the request comprises identification of the data volume, the past point in time, and a target volume in which to recreate the state of the data volume;
- determining a reference point in time representation of the data volume;
- storing the reference point in time representation in the target volume;
- obtaining a set of messages from the stream of messages for the secondary replica of the data volume, the set of messages comprising a set of write operations applied to the data volume between the reference point in time and the past point in time;
- and
- applying the set of write operations to the reference point-in-time representation of the data volume in the target volume to transform the reference point-in-time representation into a recreation of the state of the data volume at the past point in time.
6 more elements
A system comprising:
- one or more computing devices facilitating modification to a primary replica of a volume and configured with computer-executable instructions to:
- receive a set of write operations to be applied to the volume;
- synchronously apply the set of write operations to the primary replica of the volume;
- generate a set of messages to be included within a stream of messages comprising write operations to the volume;
- and
- submit the set of messages to a logging service facilitating the stream of messages;
- and
- one or more computing devices implementing the logging service and configured with computer-executable instructions to:
- receive the set of messages;
- store the set of messages within the stream of messages for delivery to one of a plurality of recipients for implementing a secondary replica of the volume;
- determine that a message exists in the message stream that has not yet been delivered to a recipient from the plurality of recipients for updating the secondary replica of the volume;
- and
- provide the message to the recipient of the plurality of recipients to cause asynchronous application of the message to the secondary replica of the volume.
10 more elements
Dependencies are inferred from the claim text.