Streaming microservices for stream processing applications

A system including one or more processors coupled to memory, the memory loaded with computer instructions to exactly-once process stream data, the instructions, when executed on the processors, implement actions comprising:

• receiving a stream of data in a queue manager;
• establishing aggregation intermediation checkpoints during processing of the data, including
• partitioning delivery of the stream data at offsets,
• saving partition demarcation offsets at the end of processing windows, and
• saving intermediate aggregation results to a distributed file system, including a window identifier (abbreviated ID), correlated offsets for the window, and intermediate results of aggregating over the window, wherein, at each checkpoint, the intermediate aggregation results are initially saved on at least one write-ahead log (abbreviated WAL) on the distributed file system and, post-saving, persisted to storage in accordance with a fault tolerance scheme;
• controlling persistence of key-value data contributing to aggregation on a partition-by-partition basis;
• and
• periodically writing out aggregations to a message queue or to a database, with the writing out governed by a fault tolerance scheme.

A non-transitory computer readable storage medium impressed with computer program instructions to exactly-once process stream data, the instructions, when executed on a processor, implement a method comprising:

• receiving a stream of data in a queue manager;
• establishing aggregation intermediation checkpoints during processing of the data, including
• partitioning delivery of the stream data at offsets,
• saving partition demarcation offsets at the end of processing windows, and
• saving intermediate aggregation results to a distributed file system, including a window identifier (abbreviated ID), correlated offsets for the window, and intermediate results of aggregating over the window, wherein, at each checkpoint, the intermediate aggregation results are initially saved on at least one write-ahead log (abbreviated WAL) on the distributed file system and, post-saving, persisted to storage in accordance with a fault tolerance scheme;
• controlling persistence of key-value data contributing to aggregation on a partition-by-partition basis;
• and
• periodically writing out aggregations to a message queue or to a database, with the writing out governed by a fault tolerance scheme.