Distributed streaming platform for real-time applications

One or more computer-readable media that are non-transitory and that store one or more programs, wherein the one or more programs, when executed, instructs one or more processors to perform the following operations:

• receive a specification for a streaming application and converting the specification into a logical plan that includes a directed acyclic graph (DAG) with operators as nodes and streams as edges, wherein the operators are program instructions and the streams are unbound sequences of streaming windows that are ordered in terms of time;
• translate the logical plan into a physical plan using stream modes specified in the specification and with one or more instances of the operators per any static partitioning in the specification, wherein at least one of the stream modes is in-rack and at least one of the stream modes is in-node and wherein streaming windows pass through a network interface controller (NIC) when the stream mode is in-rack but not when the stream mode is in-node, wherein the in-rack stream mode connects two containers of two nodes, wherein the in-node stream mode connects two containers of one node;
• obtain a plurality of containers running on a networked cluster of servers, wherein one slave executes in each container, wherein each container includes a buffer server that manages a message queue for the streaming windows;
• assign the instances to the slaves for execution according to an execution plan that depends at least in part on the physical plan and number of containers obtained, wherein each slave monitors throughput of the streaming windows through the instances in the container and other statistics related to service level and reports results of the monitoring;
• and
• make one or more dynamic adjustments based at least in part on the results of the monitoring, wherein the making of the one or more dynamic adjustments includes re-initiating the streams in accordance with a recovery policy for the application, wherein the recovery policy is one of at most once, at least once, or exactly once, wherein data loss is acceptable when the recovery policy is at most once, wherein data loss is not acceptable but extra computations are acceptable when the recovery policy is at least once, wherein data loss is not acceptable and neither are extra computations when the recovery policy is exactly once, and wherein each of the operations is executed in real time or near real time rather than offline.

A method, comprising the operations of:

• receiving a specification for a streaming application and converting the specification into a logical plan that includes a directed acyclic graph (DAG) with operators as nodes and streams as edges, wherein the operators are program instructions and the streams are unbound sequences of streaming windows that are ordered in terms of time and wherein the streaming application displays data for decision support in a dashboard that is a graphical user interface (GUI);
• translating the logical plan into a physical plan using stream modes specified in the specification and with one or more instances of the operators per any static partitioning in the specification, wherein at least one of the stream modes is in-rack and at least one of the stream modes is in-node and wherein streaming windows pass through a network interface controller (NIC) when the stream mode is in-rack but not when the stream mode is in-node, wherein the in-rack stream mode connects two containers of two nodes, wherein the in-node stream mode connects two containers of one node;
• obtaining a plurality of containers running on a networked cluster of servers, wherein one slave executes in each container, wherein each container includes a buffer server that manages a message queue for the streaming windows;
• assigning the instances to the slaves for execution according to an execution plan that depends at least in part on the physical plan and number of containers obtained, wherein each slave monitors throughput of the streaming windows through the instances in the container and other statistics related to service level and reports results of the monitoring;
• and
• making one or more dynamic adjustments based at least in part on the results of the monitoring, wherein the making of the one or more dynamic adjustments includes re-initiating the streams in accordance with a recovery policy for the application, wherein the recovery policy is one of at most once, at least once, or exactly once, wherein data loss is acceptable when the recovery policy is at most once, wherein data loss is not acceptable but extra computations are acceptable when the recovery policy is at least once, wherein data loss is not acceptable and neither are extra computations when the recovery policy is exactly once, and wherein each of the operations is executed by one or more processors in real time or near real time rather than offline.