Asynchronous in-memory data checkpointing for distributed computing systems

An article of manufacture comprising a processor-readable storage medium having stored program code which is executable by a processor to implement a method which comprises:

• maintaining, by a checkpoint manager system, global checkpoint metadata associated with asynchronous checkpoint operations that are independently performed by a plurality of operators to generate checkpoints of operator states while processing a stream of data records;
• receiving, by the checkpoint manager system, a checkpoint acknowledgment message from a given operator of the plurality of operators, wherein the checkpoint acknowledgment message comprises metadata which indicates a completion of an asynchronous checkpoint operation performed by the given operator to generate a checkpoint of the operator state that is associated with a given block of data records in the stream of data records;
• updating, by the checkpoint manager system, the global checkpoint metadata based on the metadata included in the checkpoint acknowledgment message received from the given operator;
• and
• utilizing, by the checkpoint manager system, the global checkpoint metadata to track a status of the asynchronous checkpointing operations performed by the plurality of operators with respect to the given block of data records.

A server node comprising:

• a system memory configured to store program instructions;
• and
• at least one processor configured to execute the stored program instructions in the system memory to implement a checkpoint manager system, wherein the checkpoint manager system is configured to:
• maintain global checkpoint metadata associated with asynchronous checkpoint operations that are independently performed by a plurality of operators to generate checkpoints of operator states while processing a stream of data records;
• receive a checkpoint acknowledgment message from a given operator of the plurality of operators, wherein the checkpoint acknowledgment message comprises metadata which indicates a completion of an asynchronous checkpoint operation performed by the given operator to generate a checkpoint of the operator state that is associated with a given block of data records in the stream of data records;
• update the global checkpoint metadata based on the metadata included in the checkpoint acknowledgment message received from the given operator;
• and
• utilize the global checkpoint metadata to track a status of the asynchronous checkpointing operations performed by the plurality of operators with respect to the given block of data records.