Identifying partitions in association with a data processing pipeline

Non-transitory computer readable media comprising computer-executable instructions that, when executed, cause the computing system to:

• obtain, by a streaming data processor, a specification of a data processing pipeline, wherein the specification specifies a series of nodes and interconnections between individual nodes within the series of nodes;
• implement, by the streaming data processor, the data processing pipeline by execution of a plurality of processing tasks, wherein implementing the data processing pipeline comprises:
• identifying a plurality of partitions associated with a topic of a data source from which to obtain data items for processing by the data processing pipeline;
• dynamically assigning individual partitions of the plurality of partitions to individual processing tasks of the plurality of processing tasks;
• and
• executing the plurality of processing tasks to cause individual processing tasks to retrieve data items from dynamically assigned individual partitions and process the retrieved data items according to the series of nodes and interconnections specified by the data processing pipeline.

A system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the system to:
• obtain, by a streaming data processor, a specification of a data processing pipeline, wherein the specification specifies a series of nodes and interconnections between individual nodes within the series of nodes;
• and
• implement, by the streaming data processor, the data processing pipeline by execution of a plurality of processing tasks, wherein implementing the data processing pipeline comprises:
• identifying a plurality of partitions associated with a topic of a data source from which to obtain data items for processing by the data processing pipeline;
• dynamically assigning individual partitions of the plurality of partitions to individual processing tasks of the plurality of processing tasks;
• and
• executing the plurality of processing tasks to cause individual processing tasks to retrieve data items from dynamically assigned individual partitions and process the retrieved data items according to the series of nodes and interconnections specified by the data processing pipeline.