Partitionable unifiers in distributed streaming platform for real-time applications

One or more computer-readable media that are non-transitory and that store one or more programs, wherein the one or more programs, when executed, instruct one or more processors to perform the following operations:

• receive an application that runs on a distributed streaming platform, wherein the application is structured as a directed acyclic graph (DAG) with operators as nodes and streams as edges between nodes and wherein the application includes an operator that receives a plurality of values, counts the values that are unique, and emits the unique values in a stream;
• partition an instance of the operator into at least two partitioned instances, wherein the partitioning is dynamic partitioning at run-time based on results of monitoring throughput through the instance and determining that the instance is a bottleneck;
• assign a unique value emitting from a partitioned instance to one of a plurality of unifiers according to a pre-defined scheme, wherein each unifier creates a count of the unique values received by the unifier;
• transmit the counts from each of the unifiers to a downstream instance of an operator that aggregates the counts into a sum;
• and
• display the sum in a graphical user interface (GUI), wherein the distributed streaming platform is integrated with a distributed computing framework that includes functionality for handling distributed computations that are not structured as batch jobs for Map-Reduce processing and wherein each of the operations is executed in real time or near real time rather than offline.

A method, comprising the operations of:

• receiving an application that runs on a distributed streaming platform, wherein the application is structured as a directed acyclic graph (DAG) with operators as nodes and streams as edges between nodes and wherein the application includes an operator that receives a plurality of values, counts the values that are unique, and emits the unique values in a stream;
• partitioning an instance of the operator into at least two partitioned instances, wherein the partitioning is dynamic partitioning at run-time based on results of monitoring throughput through the instance and determining that the instance is a bottleneck and the partitioning results in a round-robin distribution of values to the partitioned instances;
• assigning a unique value emitting from a partitioned instance to one of a plurality of unifiers according to a pre-defined scheme, wherein each unifier creates a count of the unique values received by the unifier;
• transmitting the counts from each of the unifiers to a downstream instance of an operator that aggregates the counts into a sum;
• and
• displaying the sum in a graphical user interface (GUI), wherein the distributed streaming platform is integrated with a distributed computing framework that includes functionality for handling distributed computations that are not structured as batch jobs for Map-Reduce processing and wherein each of the operations is executed by one or more processors in real time or near real time rather than offline.