Continuous flow checkpointing data processing

A method for continuous flow checkpointing in a data processing system having one or more sources for receiving and storing input data, one or more processes for receiving and processing data from one or more sources or prior processes, and one or more sinks for receiving processed data from one or more processes or sources and for publishing processed data, the method including:

• (a) transmitting a checkpoint request message to every source;
• (b) suspending normal data processing in each source in response to receipt of such checkpoint request message, saving a current checkpoint record sufficient to reconstruct the state of such source, propagating a checkpoint message from such source to any process that consumes data from such source, and resuming normal data processing in each source;
• (c) suspending normal data processing in each process in response to receiving checkpoint messages from every source or prior process from which such process consumes data, saving a current checkpoint record sufficient to reconstruct the state of such process, propagating the checkpoint message from such process to any process or sink that consumes data from such process, and resuming normal data processing in such process;
• (d) suspending normal data processing in each sink in response to receiving checkpoint messages from every process from which such sink consumes data, saving a current checkpoint record sufficient to reconstruct the state of such sink, saving any unpublished data, and resuming normal data processing in such sink.

A method for continuous flow checkpointing in a data processing system having one or more sources for receiving and storing input data, one or more processes for receiving and processing data from one or more sources or prior processes, and one or more sinks for receiving processed data from one or more processes or sources and for publishing processed data, the method including:

• (a) transmitting a checkpoint request message to every source;
• (b) suspending normal data processing in each source in response to receipt of such checkpoint request message, saving a current checkpoint record sufficient to reconstruct the state of such source, propagating a checkpoint message from such source to any process that consumes data from such source, and resuming normal data processing in each source;
• (c) suspending normal data processing in each process in response to receiving checkpoint messages from every source or prior process from which such process consumes data, saving a current checkpoint record sufficient to reconstruct the state of such process, propagating the checkpoint message from such process to any process or sink that consumes data from such process, and resuming normal data processing in such process;
• (d) suspending normal data processing in each sink in response to receiving checkpoint messages from every process from which such sink consumes data, saving a current checkpoint record sufficient to reconstruct the state of such sink, saving any unpublished data, and propagating the checkpoint message from each sink to a checkpoint processor;
• (e) receiving the checkpoint messages from all sinks, and in response to such receipt, updating a stored value indicating completion of checkpointing in all sources, processes, and sinks, and transmitting the stored value to each sink;
• and
• (f) receiving the stored value in each sink and, in response to such receipt, publishing any unpublished data associated with such sink and resuming normal data processing in such sink.

A computer program, stored on a computer-readable medium, for continuous flow checkpointing in a data processing system having one or more sources for receiving and storing input data, one or more processes for receiving and processing data from one or more sources or prior processes, and one or more sinks for receiving processed data from one or more processes or sources and for publishing processed data, the computer program comprising instructions for causing a computer to:

• (a) transmit a checkpoint request message to every source;
• (b) suspend normal data processing in each source in response to receipt of such checkpoint request message, save a current checkpoint record sufficient to reconstruct the state of such source, propagate a checkpoint message from such source to any process that consumes data from such source, and resume normal data processing in each source;
• (c) suspend normal data processing in each process in response to receiving checkpoint messages from every source or prior process from which such process consumes data, save a current checkpoint record sufficient to reconstruct the state of such process, propagate the checkpoint message from such process to any process or sink that consumes data from such process, and resume normal data processing in such process;
• (d) suspend normal data processing in each sink in response to receiving checkpoint messages from every process from which such sink consumes data, save a current checkpoint record sufficient to reconstruct the state of such sink, save any unpublished data, and resume normal data processing in such sink.

A computer program, stored on a computer-readable medium, for continuous flow checkpointing in a data processing system having one or more sources for receiving and storing input data, one or more processes for receiving and processing data from one or more sources or prior processes, and one or more sinks for receiving processed data from one or more processes or sources and for publishing processed data, the computer program comprising instructions for causing a computer to:

• (a) transmit a checkpoint request message to every source;
• (b) suspend normal data processing in each source in response to receipt of such checkpoint request message, save a current checkpoint record sufficient to reconstruct the state of such source, propagate a checkpoint message from such source to any process that consumes data from such source, and resume normal data processing in each source;
• (c) suspend normal data processing in each process in response to receiving checkpoint messages from every source or prior process from which such process consumes data, save a current checkpoint record sufficient to reconstruct the state of such process, propagate the checkpoint message from such process to any process or sink that consumes data from such process, and resume normal data processing in such process;
• (d) suspend normal data processing in each sink in response to receiving checkpoint messages from every process from which such sink consumes data, save a current checkpoint record sufficient to reconstruct the state of such sink, save any unpublished data, and propagate the checkpoint message from each sink to a checkpoint processor;
• (e) receive the checkpoint messages from all sinks, and in response to such receipt, update a stored value indicating completion of checkpointing in all sources, processes, and sinks, and transmit the stored value to each sink;
• and
• (f) receive the stored value in each sink and, in response to such receipt, publish any unpublished data associated with such sink and resume normal data processing in such sink.