US 2026/0288595 A1 · REPLICATION RECOVERY TECHNIQUES FOR SIDELINED PARTITIONS

Abstract

The disclosed systems, methods, and computer readable media relate to sidelining a data stream partition (e.g., halting replication of the partition) and resuming replication for a sidelined partition after an a poison pill event that triggered the sidelining is resolved. While the partition is sidelined, stream event processing for the partition may be restricted/blocked/halted, while stream event processing for active partitions of the data stream continue. After the error is resolved, a process for resynchronizing the partition based on the stream events that were received from the point at which the error was originally detected may be performed. Some stream events for the partition may be obtained from a snapshot generated by a snapshot service, while other stream events during the resynchronization may be obtained from a demuxer. Once the partition is resynchronized, the partition may be un-sidelined such that subsequent stream events for the partition may be processed unrestricted.

Claims20

Dependent claims

A method, comprising:

receiving, by a client device from a demuxer of a replication streaming system, a first stream event corresponding to a data stream that is associated with a plurality of partitions, the first stream event identifying a partition of the plurality of partitions and a streaming cursor, the client device tracking unrestricted replication for the partition using the streaming cursor;
in response to detecting an error corresponding to the first stream event, restricting, by the client device, replication of the partition based on generating a sideline cursor from the streaming cursor;
executing, by the client device using the sideline cursor, a bootstrap process during which a second stream event for the partition is obtained from a snapshot of the data stream;
executing, by the client device using the sideline cursor, a catch up process during which a third stream event occurring after the second stream event in the data stream is obtained;
and
resuming, by the client device, the unrestricted replication for the partition based at least in part on concluding the catch up process.

3 more elements

A client device, comprising:

one or more processors;
and
one or more memories storing computer-executable instructions that, when executed by the one or more processors, causes the one or more processors to:
incrementally process stream events of a data stream that is associated with a plurality of partitions, a most-recently received stream event identifying a partition of the plurality of partitions and a first cursor that represents a position in the data stream corresponding to the most-recently received stream event;
in response to detecting an error corresponding to the most-recently received stream event, restrict replication at the partition;
obtain, using a second cursor that indicates the position in the data stream corresponding to the most-recently received stream event, a first set of stream events for the partition, the first set of stream events being obtained from a snapshot of the data stream;
obtain a second set of stream events for the partition, the second set of stream events occurring after the first set of stream events in the data stream;
and
resume unrestricted replication of the partition based at least in part on obtaining the first set of stream events and the second set of stream events.

6 more elements

A non-transitory computer-readable medium comprising one or more memories storing computer-executable instructions that, when executed by one or more processors of a client device, cause the one or more processors to:

incrementally process stream events of a data stream that is associated with a plurality of partitions;
track processing of the stream events based at least in part on maintaining a first set of cursors that individually indicate a last stream event that was successfully processed for a respective partition;
while continuing to process stream events for the data stream that correspond to at least one partition of the plurality of partitions, restrict stream event processing for a second partition of the plurality of partitions based at least in part on detecting an error associated with a stream event associated with the second partition;
responsive to determining that the stream event processing for the second partition is to resume, obtain a set of stream events for the second partition, the set of stream events occurring in the data stream after the stream event associated with the error;
track, using a second cursor that is separate from the first set of cursors, processing of the set of stream events for the second partition;
and
resume unrestricted replication of the partition based at least in part on tracking the processing of the set of stream events for the second partition using the second cursor.

4 more elements

Dependencies are inferred from the claim text.

Replication recovery techniques for sidelined partitions

Life of this patent

Replication recovery techniques for sidelined partitions

Life of this patent

Abstract

Claims20