Stream processing at scale

A method comprising:

• connecting two or more instances as a topology, wherein at least one of the instances is a spout and at least one of the instances is a bolt, wherein each spout transfers tuples to one or more bolts and wherein each bolt performs a computation on the transferred tuples;
• submitting the topology to a scheduler, wherein the service scheduler receives resource offers from a cluster manager representing computing resources available on one or more of cluster nodes and determines resources to accept and computations to run on the accepted computing resources;
• scheduling the topology as one or more jobs, wherein each job includes two or more containers, including a first container and a second container, the first container including a topology master and the second container including a stream manager and one or more stream processing system (SPS) instances, wherein each SPS instance represents one of the instances in the topology;
• executing the topology master as one or more tasks on one or more of the cluster nodes;
• executing the stream manager as one or more tasks on one or more of the cluster nodes;
• executing each SPS instance as one or more tasks on one or more of the cluster nodes,
• detecting a failure of the stream manager;
• and
• responsive to detecting the failure of the stream manager, restarting the stream manager in the second container, wherein restarting includes rediscovering the topology master and retrieving a copy of a physical plan from the topology master.

A non-transitory computer-readable storage medium encoded with instructions that, when executed, cause at least one processor of a computing device to:

• connect two or more instances as a topology, wherein at least one of the instance is a spout and at least one of the instances is a bolt, wherein each spout transfers tuples to one or more bolts and wherein each bolt performs a computation on the transferred tuples;
• submit a topology to a scheduler, wherein the topology connects two or more instances as a directed graph and wherein the service scheduler receives resource offers from a cluster manager representing computing resources available on one or more of cluster nodes and determines resources to accept and computations to run on the accepted computing resources;
• schedule the topology as one or more jobs, wherein each job includes two or more containers, including a first container and a second container, the first container including a topology master and the second container including a stream manager and one or more stream processing system (SPS) instances, wherein each SPS instance represents one of the instances in the topology;
• execute the topology master as one or more tasks on the one or more of cluster nodes;
• execute the stream manager as one or more tasks on the one or more of cluster nodes;
• execute each SPS instance as one or more tasks on the one or more of cluster nodes,
• detect a failure of the stream manager;
• and
• responsive to detecting the failure of the stream manager, restart the stream manager in the second container, wherein the restart includes rediscovering the topology master and retrieving a copy of a physical plan from the topology master.

A method comprising:

• connecting two or more instances as a topology, wherein at least one of the instances is a spout and at least one of the instances is a bolt, wherein each spout transfers tuples to one or more bolts and wherein each bolt performs a computation on the transferred tuples;
• submitting the topology to a scheduler, wherein the service scheduler receives resource offers from a cluster manager representing computing resources available on one or more of cluster nodes and determines resources to accept and computations to run on the accepted computing resources;
• scheduling the topology as one or more jobs, wherein each job includes two or more containers, including a first container and a second container, the first container including a topology master and the second container including a stream manager and one or more stream processing system (SPS) instances, wherein each SPS instance represents one of the instances in the topology;
• executing the topology master as one or more tasks on one or more of the cluster nodes;
• executing the stream manager as one or more tasks on one or more of the cluster nodes;
• executing each SPS instance as one or more tasks on one or more of the cluster nodes,
• detecting a failure of an SPS instance of the SPS instances;
• and
• responsive to detecting the failure of the SPS instance, restarting the SPS instance in the second container, wherein restarting includes contacting the stream manager in the second container, retrieving a copy of a physical plan, and executing user code corresponding to the physical plan.

A non-transitory computer-readable storage medium encoded with instructions that, when executed, cause at least one processor of a computing device to:

• connect two or more instances as a topology, wherein at least one of the instance is a spout and at least one of the instances is a bolt, wherein each spout transfers tuples to one or more bolts and wherein each bolt performs a computation on the transferred tuples;
• submit a topology to a scheduler, wherein the topology connects two or more instances as a directed graph and wherein the service scheduler receives resource offers from a cluster manager representing computing resources available on one or more of cluster nodes and determines resources to accept and computations to run on the accepted computing resources;
• schedule the topology as one or more jobs, wherein each job includes two or more containers, including a first container and a second container, the first container including a topology master and the second container including a stream manager and one or more stream processing system (SPS) instances, wherein each SPS instance represents one of the instances in the topology;
• execute the topology master as one or more tasks on the one or more of cluster nodes;
• execute the stream manager as one or more tasks on the one or more of cluster nodes;
• execute each SPS instance as one or more tasks on the one or more of cluster nodes,
• detecting a failure of an SPS instance of the SPS instances;
• and
• responsive to detecting the failure of the SPS instance, restart the SPS instance in the second container, wherein the restart includes contacting the stream manager in the second container, retrieving a copy of a physical plan, and executing user code corresponding to the physical plan.