Fault-tolerant stream processing

A system comprising:

• a data processing system that includes one or more processors and non-transitory machine readable storage medium having instructions stored thereon that when executed by the one or more processors cause the one or more processors to perform the process comprising:
• processing, at the data processing system, input events to generate primary output events and secondary output events, wherein the primary output events are generated by a primary server of the data processing system and the secondary output events are generated by one or more secondary servers of the data processing system;
• writing, by the data processing system, the primary output events to a primary target and the secondary output events to one or more secondary targets, wherein the primary output events are written by the primary server and the secondary output events are written by the one or more secondary servers;
• receiving, at the data processing system, an election of a new primary server from a synchronization system upon a failure of the primary server, wherein the new primary server is elected from the one or more secondary servers;
• reading, by the new primary server of the data processing system, the secondary output events from the one or more secondary targets;
• determining, by the new primary server of the data processing system, the primary output events that failed to be written to the primary target because of the failure of the primary server;
• and
• writing, by the new primary server of the data processing system, the failed primary output events to the primary target using the secondary output events read from the one or more secondary targets.

A non-transitory machine readable storage medium having instructions stored thereon that when executed by one or more processors cause the one or more processors to perform a method comprising:

• processing input events to generate primary output events and secondary output events, wherein the primary output events are generated by a primary server of a data processing system and the secondary output events are generated by one or more secondary servers of the data processing system;
• writing the primary output events to a primary target and the secondary output events to one or more secondary targets, wherein the primary output events are written by the primary server and the secondary output events are written by the one or more secondary servers;
• receiving an election of a new primary server from a synchronization system upon a failure of the primary server, wherein the new primary server is elected from the one or more secondary servers;
• reading, by the new primary server, the secondary output events from the one or more secondary targets;
• determining, by the new primary server, the primary output events that failed to be written to the primary target because of the failure of the primary server;
• and
• writing, by the new primary server, the failed primary output events to the primary target using the secondary output events read from the one or more secondary targets.