Checkpointing in Distributed Streaming Platform for Real-Time Applications

A non-transitory computer-readable storage medium storing a computer program, the computer-readable medium comprising:

• program instructions for receiving a data stream for an application;
• program instructions for converting the data stream into a plurality of data tuples structured according to a schema;
• program instructions for emitting the plurality of data tuples as a plurality of streaming windows, each of the plurality of streaming windows being separated from other streaming windows by a leading control tuple and a trailing control tuple, both the leading control tuple and the trailing control tuple being associated with a same ordinal identifier specific to each respective streaming window;
• and
• program instructions for emitting a checkpointing tuple, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed in real time or near real time rather than offline.

A method, comprising:

• receiving a data stream for an application, the application using one or more stream modes from the group of stream modes consisting of in-node and in-rack;
• converting the data stream into a plurality of data tuples structured according to a schema;
• emitting the plurality of data tuples as a plurality of streaming windows, which is separated from other streaming windows by a leading control tuple and a trailing control tuple associated with a same ordinal identifier specific to each respective streaming window;
• and
• emitting a checkpointing tuple, wherein the checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance, wherein each of the operations is executed by one or more processors in real time or near real time rather than offline.