Dynamic flow control for stream processing

A computing system comprising:

• a memory configured to store program instructions;
• and
• a processor configured to execute the program instructions, wherein, when executed, the program instructions cause the processor to
• execute a stream processing application which reads data from a data source and writes the data to a target destination, the stream processing application comprising a pipeline of operators which perform different processing operations on the data,
• determine that a runtime processing speed of at least two operators among the pipeline of operators included in the stream processing application are out of sync with respect to each other,
• determining an average speed of the data stream processing application processing a write request from the data source to the target destination, and
• dynamically synchronize the runtime processing speed of the at least two out of sync operators in the pipeline of operators of the stream processing application by changing a processing speed of at least one operator based on an overall processing performance of the stream processing application as a whole,
• wherein the dynamically synchronizing comprises, in response to the average speed exceeding a predetermined upper limit, slowing down the processing speed of the at least one operator, and in response to the average speed being below a predetermined lower limit, speeding up the processing speed of the at least one operator.

A non-transitory computer readable medium having stored therein instructions that when executed cause a computer to perform a method comprising:

• executing a stream processing application which reads data from a data source and writes the data to a target destination, the stream processing application comprising a pipeline of operators which perform different processing operations on the data;
• determining that a runtime processing speed of at least two operators among the pipeline of operators included in the stream processing application are out of sync with respect to each other;
• determining an average speed of the data stream processing application processing a write request from the data source to the target destination;
• and
• dynamically synchronizing the runtime processing speed of the at least two out of sync operators in the pipeline of operators of the stream processing application by changing a processing speed of at least one operator based on an overall processing performance of the stream processing application as a whole,
• wherein the dynamically synchronizing comprises, in response to the average speed exceeding a predetermined upper limit, slowing down the processing speed of the at least one operator, and in response to the average speed being below a predetermined lower limit, speeding up the processing speed of the at least one operator.