US 11,599,293 B2Granted

Consistent data stream replication and reconstruction in a streaming data storage platform

AssigneeEMC IP Holding Company LLC

InventorsAndrei Paduroiu, Igor Medvedev

Life of this patent

Estimating the term…

Loading patent…

Abstract

The described technology is generally directed towards consistently replicating and reconstructing a data stream made up of a dynamic set of (ordered) segments into a different location (e.g., cluster) from the one in which the stream was created. The technology facilitates consistently and generally continuously and replicating a stream of events ingested in a source cluster to a target cluster for consumption (reading). As stream data segments are replicated to a target cluster by a replicator which is not guaranteed to keep the replicated data consistent, a target controller reconstructs the replicated data stream up to a stream cut point at which the replicated data has been sufficiently replicated so as to be consistent. Reading of the replicated data stream is limited to a view up to the stream cut point; as more data is replicated, additional data up to a later stream cut point becomes available for reading.

Claims20

Dependent claims

A system, comprising:

a processor;
and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, the operations comprising:
obtaining part of a data stream at a target cluster of a streaming data storage system via replication from a source cluster of the streaming data storage system, the data stream comprising events maintained in segments of the data stream, wherein the segments are parallel segments separating the data stream according to routing key values;
monitoring stream cut data, the stream cut data comprising stream cuts made to the data stream, wherein the stream cuts comprise respective groups of segment identifiers that identify respective segments of respective groups of segments of the stream cuts and respective segment lengths of the respective segments of the respective groups of segments of the stream cuts, and wherein the stream cut data is stored via a stream cut segment of the data stream that is parallel to a segment of the segments;
determining, from the stream cut data and current lengths of the segments obtained via the replication, a most recent stream cut point to which the segments have been fully replicated;
and
limiting reading of the segments to events before the most recent stream cut point to which the segments have been fully replicated.

5 more elements

A method, comprising:

obtaining, via a processor of a target cluster, replicated segments comprising streamed data of a data stream from a source cluster;
obtaining a stream cut comprising identifiers for respective segments of the data stream and offset values representing lengths of the respective segments relative to the stream cut wherein the segments are parallel segments separating the data stream according to routing key values, and wherein the stream cut is comprised in an event of a stream cut segment that is a parallel segment to the parallel segments of the data stream;
determining whether current lengths of the replicated segments are greater than or equal to corresponding lengths represented by the offset values in the stream cut;
and
in response to determining that the current lengths of the replicated segments are greater than or equal to the offset values in the stream cut,
updating target offset data of the replicated segments to match the offset values in the stream cut, resulting in updated target offset data, and
allowing reading of the streamed data from the replicated segments up to locations in the replicated segments represented by the updated target offset data.

4 more elements

A non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor of a streaming data storage system, facilitate performance of operations, the operations comprising:

receiving, at a target cluster, segment data of segments of a data stream being replicated from a source cluster, wherein the segment data of the segments is not guaranteed to be consistent during replication, and wherein the segments of the data stream are parallel segments separating the data stream according to routing key values;
determining a selected stream cut point at which the segment data replicated from the source cluster is consistent among the segments, wherein the stream cut point is comprised in a stream cut event of a stream cut segment that is a parallel segment to the parallel segments of the data stream;
and
presenting a view of the data stream to a reader of the target cluster, in which the view is limited to the segment data before the selected stream cut point.

1 more element

Dependencies are inferred from the claim text.

Facts

Application: 17/070,029Filed 14 Oct 2020
Published: US 2022/0113871 A114 Apr 2022
Granted: 7 Mar 2023
Term ends: Not estimatedThe source records don’t support an estimate.

Classification9 codes

G06F 3/065
Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems: replication mechanisms
G06F 3/0685
Plurality of storage devices: hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
G06F 3/0604
Specifically adapted to achieve a particular effect: improving or facilitating administration, e.g. storage management
G06F 3/061
Specifically adapted to achieve a particular effect: improving I/O performance