Dynamic partitioning of instances in distributed streaming platform for real-time applications

One or more computer-readable media that are non-transitory and that store one or more programs, wherein the one or more programs, when executed, instructs one or more processors to perform the following operations:

• monitor a first operator instance of a first operator on an upstream node for an application running on a distributed streaming platform over a networked cluster of servers, wherein the first operator instance includes an input port that receives streaming windows from an upstream instance and an output port that emits streaming windows to a second operator instance of a second operator on a downstream node, wherein the monitoring includes determining a window identifier for a streaming window which the first operator instance is processing, and wherein each streaming window is an atomic sequence of tuples associated with a recovery policy;
• report results of the monitoring to a master;
• receive a dynamic adjustment from the master, wherein the dynamic adjustment includes launching a plurality of additional operator instances of the first operator and using cascading unifiers to connect them and the first operator instance through a plurality of network interface controllers (NICs) to the second operator instance and wherein the plurality of additional operator instances is three additional operator instances;
• and
• resume a flow of streaming windows into the input port and the output port according to the recovery policy for the first operator instance, wherein the recovery policy is at least once, wherein data loss is not acceptable but extra computations are acceptable and checkpointing of an operator instance does not occur after every streaming window when the recovery policy is at least once, and wherein each of the operations is executed in real time or near real time rather than offline.

A method, comprising:

• monitoring a first operator instance of a first operator on an upstream node for an application running on a distributed streaming platform over a networked cluster of servers, wherein the first operator instance includes an input port that receives streaming windows from an upstream instance and an output port that emits streaming windows to a second operator instance of a second operator on a downstream node, wherein the monitoring includes determining a window identifier for a streaming window which the first operator instance is processing, and wherein each streaming window is an atomic sequence of tuples associated with a recovery policy;
• reporting results of the monitoring to a master;
• receiving a dynamic adjustment from the master, wherein the dynamic adjustment includes launching a plurality of additional operator instances of the first operator and using cascading unifiers to connect them and the first operator instance through a plurality of network interface controllers (NICs) to the second operator instance and wherein the plurality of additional operator instances is three additional operator instances;
• and
• resuming a flow of streaming windows into the input port and the output port according to the recovery policy for the first operator instance, wherein the recovery policy is at least once, wherein data loss is not acceptable but extra computations are acceptable and checkpointing of an operator instance does not occur after every streaming window when the recovery policy is at least once, and wherein each of the operations is executed by one or more processors in real time or near real time rather than offline.