Recovery and fault-tolerance under computational indeterminism

Software stored in a non-transitory form on a computer-readable medium, for promoting fault tolerance and recovery in a computing system including at least one processing node, the software including instructions for causing a computing system to:

• promote availability and recovery of a first processing node, wherein promoting availability and recovery includes, at a first processing node,
• executing a spawner at said node, wherein said spawner, in the course of execution, generates a first spawn,
• wherein executing said spawner includes assigning, to said spawner, a first generation indicator,
• wherein said first spawn inherits said first generation indicator;
• beginning a checkpoint interval, at the end of which nodal recovery information, which is usable for recovery of said node, is committed to durable storage, wherein beginning said checkpoint interval includes
• suspending said spawner from generating spawn,
• assigning, to said spawner, a second generation indicator that differs from said first generation indicator,
• resuming said spawner, thereby enabling said spawner to generate a second spawn, wherein said second spawn inherits said second generation indicator, and
• controlling an extent to which said second spawn writes to memory;
• and
• after committing said nodal recovery information, releasing control over said extent to which said second spawn can write to memory.

A computing system including:

• a data storage system including durable storage;
• and
• one or more processing nodes including least one processor configured to promote availability and recovery of a first processing node, wherein promoting availability and recovery includes, at a first processing node,
• executing a spawner at said node, wherein said spawner, in the course of execution, generates a first spawn,
• wherein executing said spawner includes assigning, to said spawner, a first generation indicator,
• wherein said first spawn inherits said first generation indicator;
• beginning a checkpoint interval, at the end of which nodal recovery information, which is usable for recovery of said node, is committed to durable storage, wherein beginning said checkpoint interval includes
• suspending said spawner from generating spawn,
• assigning, to said spawner, a second generation indicator that differs from said first generation indicator,
• resuming said spawner, thereby enabling said spawner to generate a second spawn, wherein said second spawn inherits said second generation indicator, and
• controlling an extent to which said second spawn writes to memory;
• and
• after committing said nodal recovery information, releasing control over said extent to which said second spawn can write to memory.

An apparatus including:

• means for durably storing data in durable storage;
• and
• means for promoting availability and recovery of a first processing node, wherein promoting availability and recovery includes, at a first processing node,
• executing a spawner at said node, wherein said spawner, in the course of execution, generates a first spawn,
• wherein executing said spawner includes assigning, to said spawner, a first generation indicator,
• wherein said first spawn inherits said first generation indicator;
• beginning a checkpoint interval, at the end of which nodal recovery information, which is usable for recovery of said node, is committed to durable storage, wherein beginning said checkpoint interval includes
• suspending said spawner from generating spawn,
• assigning, to said spawner, a second generation indicator that differs from said first generation indicator,
• resuming said spawner, thereby enabling said spawner to generate a second spawn, wherein said second spawn inherits said second generation indicator, and
• controlling an extent to which said second spawn writes to memory;
• and
• after committing said nodal recovery information, releasing control over said extent to which said second spawn can write to memory.