Failure recovery for stream processing applications

A non-transitory computer readable storage device containing an executable program for providing a failure recovery mechanism for a stream processing application comprising a plurality of components, where the program performs the steps of:

• receiving, by a compiler, source code for the stream processing application, wherein the source code defines a checkpointing frequency with which a checkpoint file is to be generated for each of the plurality of components, and wherein respective checkpointing frequencies defined for at least two of the plurality of components are different, and wherein different respective checkpointing frequencies are defined for the at least two of the plurality of components based on a parameter that differs between the at least two of the plurality of components, the parameter being at least one of:
• whether the at least two of the plurality of components are stateful or stateless, whether the at least two of the plurality of components are windowed stream operators, or sizes of respective states of the at least two of the plurality of components, wherein a stateful component is a component that operates on state variables, and wherein a stateless component is a component that does not operate on state variables;
• generating, by the compiler, a sequence of instructions for converting a state of the at least one of the plurality of components into a checkpoint file comprising a sequence of storable bits on a periodic basis, according to the checkpointing frequency defined for the at least one of the plurality of components;
• initiating execution of the stream processing application;
• and
• storing, during execution of the stream processing application, the checkpoint file at a location that is accessible after failure recovery.

An apparatus for providing a failure recovery mechanism for a stream processing application comprising a plurality of components, the apparatus comprising:

• a processor;
• a memory;
• an input device to receive source code for the stream processing application, where the source code defines a subset of the plurality of components that are fault tolerant, and wherein at least two components in the subset are associated with a different checkpointing frequency that defines a frequency with which a checkpoint file is to be generated for an associated one of the at least two components, and wherein different respective checkpointing frequencies are defined for the at least two of the plurality of components based on a parameter that differs between the at least two of the plurality of components, the parameter being at least one of:
• whether the at least two of the plurality of components are stateful or stateless, whether the at least two of the plurality of components are windowed stream operators, or sizes of respective states of the at least two of the plurality of components, wherein a stateful component is a component that operates on state variables, and wherein a stateless component is a component that does not operate on state variables;
• a compiler to translate the source code into specialized code executable on a target machine, where the specialized code defines one or more methods for providing fault tolerance to the subset of the plurality of components;
• and
• an output device to provide the specialized code to the target machine.