Distributed Streaming Platform for Real-time Applications

A non-transitory computer-readable medium storing a computer program for implementing a method, the computer-readable medium comprising:

• program instructions for receiving a specification for a streaming application and converting the specification into a logical plan that includes a directed acyclic graph (DAG) with operators as nodes and streams as edges, wherein the operators are program instructions and the streams are unbound sequences of streaming windows that are ordered in terms of time;
• program instructions for translating the logical plan into a physical plan using stream modes specified in the specification and with one or more instances of the operators per any static partitioning in the specification, wherein at least one of the stream modes is in-rack and at least one of the stream modes is in-node;
• program instructions for obtaining a plurality of containers running on a networked cluster of servers, wherein one slave executes in each container, wherein each container includes a buffer server that manages a message queue for the streaming windows;
• program instructions for assigning the instances to the slaves for execution according to an execution plan that depends at least in part on the physical plan and number of containers obtained, wherein each slave monitors throughput of the streaming windows through the instances in the corresponding container and other statistics related to service level and reports results of the monitoring;
• program instructions for launching the execution plan by converting external input into tuples grouped into streaming windows;
• program instructions for determining an instance of a first operator of a first container is creating a bottleneck based on the throughput monitoring by each slave;
• program instructions for creating multiple instances of the first operator and assigning the multiple instances to the first container;
• program instructions for hashing each tuple of a first streaming window for the first operator using a hash function;
• and
• program instructions for assigning each tuple of a first streaming window for the first operator to one of the multiple instances based on selected bits of a corresponding hash code generated by the hashing.

A computer system comprising:

• a processor;
• and
• memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method comprising:
• receiving a specification for a streaming application and converting the specification into a logical plan that includes a directed acyclic graph (DAG) with operators as nodes and streams as edges, wherein the operators are program instructions and the streams are unbound sequences of streaming windows that are ordered in terms of time;
• translating the logical plan into a physical plan using stream modes specified in the specification and with one or more instances of the operators per any static partitioning in the specification, wherein at least one of the stream modes is in-rack and at least one of the stream modes is in-node;
• obtaining a plurality of containers running on a networked cluster of servers, wherein one slave executes in each container, wherein each container includes a buffer server that manages a message queue for the streaming windows;
• assigning the instances to the slaves for execution according to an execution plan that depends at least in part on the physical plan and number of containers obtained, wherein each slave monitors throughput of the streaming windows through the instances in the corresponding container and other statistics related to service level and reports results of the monitoring;
• launching the execution plan by converting external input into tuples grouped into streaming windows;
• determining an instance of a first operator of a first container is creating a bottleneck based on the throughput monitoring by each slave;
• creating multiple instances of the first operator and assigning the multiple instances to the first container;
• hashing each tuple of a first streaming window for the first operator using a hash function;
• and
• assigning each tuple of a first streaming window for the first operator to one of the multiple instances based on selected bits of a corresponding hash code generated by the hashing.