Method for high-performance data stream processing

A computer program product comprising a non-transitory computer readable memory having computer readable program code for optimizing data stream processing, said computer program product including:

• computer readable program code for employing a split, aggregate and join architectural pattern on an application, wherein the pattern processes the application in one or more different domains;
• computer readable program code for employing multiple per-group operators to perform multiple simultaneous group-independent aggregation, split operations and join operation on one or more incoming streams of the application via (i) de-multiplexing the data from the one or more incoming streams into multiple physical streams, the one or more incoming streams containing data belonging to multiple groups multiplexed together with a number of groups is not known a priori and each physical stream corresponding to one of the multiple groups, (ii) aggregating data from the multiple groups to form multiple aggregates and (iii) joining the multiple aggregates with additional data incoming from a same or one and more other groups, and wherein said employing the multiple per-group operators comprises applying a distinct window for performing the simultaneous group-independent split, aggregation and join operations for each of the multiple groups, wherein each said distinct window triggers performance of said operations and termination of the performance of said operations based on the tuples that have been accumulated for the corresponding group;
• computer readable program code for generating code, wherein the code facilitates mapping of the application onto a computational infrastructure to enable workload partitioning;
• computer readable program code for using the one or more per-group operators to decompose the application into one or more granular components;
• computer readable program code for using the code to reassemble the one or more granular components into one or more deployable blocks to map the application to the computational infrastructure, wherein reassembling the one or more granular components to map the application to the computational infrastructure optimizes data stream processing of the application;
• and
• computer readable program code for refining the level of granularity associated with the one or more granular components based on said computational infrastructure.

An apparatus for optimizing data stream processing, comprising:

• a memory;
• and
• at least one processor coupled to said memory and operative to:
• employ a split, aggregate and join architectural pattern on an application, wherein the pattern processes the application in one or more different domains;
• employ multiple per-group operators to perform multiple simultaneous group-independent aggregation, split operations and join operation on one or more incoming streams of the application via (i) de-multiplexing the data from the one or more incoming streams into multiple physical streams, the one or more incoming streams containing data belonging to multiple groups multiplexed together with a number of groups is not known a priori and each physical stream corresponding to one of the multiple groups, (ii) aggregating data from the multiple groups to form multiple aggregates and (iii) joining the multiple aggregates with additional data incoming from a same or one and more other groups, and wherein said employing the multiple per-group operators comprises applying a distinct window for performing the simultaneous group-independent split, aggregation and join operations for each of the multiple groups, wherein each said distinct window triggers performance of said operations and termination of the performance of said operations based on the tuples that have been accumulated for the corresponding group;
• generate code, wherein the code facilitates mapping of the application onto a computational infrastructure to enable workload partitioning;
• use the one or more per-group operators to decompose the application into one or more granular components;
• use the code to reassemble the one or more granular components into one or more deployable blocks to map the application to the computational infrastructure, wherein reassembling the one or more granular components to map the application to the computational infrastructure optimizes data stream processing of the application;
• and
• refine the level of granularity associated with the one or more granular components based on said computational infrastructure.