Systems and methods for failure recovery in at-most-once and exactly-once streaming data processing

A non-transitory computer readable medium holding program instructions for execution on one or more hardware processors of one or more computers, the program instructions, when executed on the one or more hardware processors:

• providing a plurality of sending nodes, a first node, and, a second node, where the plurality of sending nodes contemporaneously stream input data chunks to the first node;
• causing the first node to generate one or more output data chunks from the input data chunks from the plurality of sending nodes, and the first node streaming the one or more output data chunks to the second node;
• causing the first node to communicate with each of the plurality of sending nodes using a protocol in which transfer of each input data chunk from a respective sending node is concluded by a message exchange that commits the first node to perform failure recovery with respect to handling the just-transferred input data chunk, and releases the respective sending node therefrom;
• and,
• causing the first node to begin sending the one or more output data chunks prior to receiving all of the input data chunks associated with the one or more output data chunks;
• and, wherein said failure recovery by the first node comprises:
• upon a failure prior to a commitment for a given input data chunk from a given one of the plurality of sending nodes, the first node:
• sending a failure message to the second node with respect to the one or more output data chunks,
• discarding a received portion of the given input data chunk that failed,
• re-generating the one or more output data chunks without the given input data chunk incorporated therein, based on a copy of the one or more input data chunks other than the given input data chunk that failed, the copy being stored at the first node, and,
• sending the re-generated one or more output data chunks to the second node.