Distributed fault-tolerance via disaggregated memory boards

A memory device of a distributed computing system, the memory device comprising:

• multiple task queues that store task operations performed by multiple compute nodes of the distributed computing system;
• memory that stores snapshots received from the multiple compute nodes, each snapshot including a task database instance;
• and
• a memory controller in communication with the multiple task queues and the memory, the memory controller configured to:
• set a current checkpoint when the received snapshots match and store a task database instance corresponding to the current checkpoint;
• set the task database to a previous checkpoint when a mismatch in the received snapshots has been detected;
• distribute a correct checkpoint task database instance to at least one compute node of the multiple compute nodes in response to detecting the mismatch in the snapshots;
• receive performance data of time to complete the task for the multiple compute nodes;
• identify at least one lagging compute node of the multiple compute nodes during the task using the performance data;
• not use a snapshot received from the at least one lagging compute node when determining whether the received snapshots match;
• determine likelihood of a timing fault of at the least one lagging compute node in performing the task operations;
• and
• send an indication of the determined likelihood of the timing fault to a task server of the distributed computing system;
• and
• wherein the task server changes the combination of the compute nodes that are providing results for the task according to the determined likelihood.

A distributed computing system, the distributed computing system comprising:

• multiple compute nodes, a combination of the multiple compute nodes configured to execute a task and determine performance metrics related to the task;
• a memory node in communication with the multiple compute nodes and separate from the compute nodes, the memory node configured to receive the performance metrics from the combination of compute nodes;
• a task server in communication with the multiple compute nodes and configured to:
• determine performance data of time to complete the task for compute nodes in the combination of compute nodes;
• identify at least one lagging compute node in the combination of compute nodes during the task using the performance data;
• determine likelihood of a timing fault in completing the task by the at least one lagging compute node during the task using the performance data;
• change the combination of compute nodes providing results for the task according to the determined likelihood of a timing fault of at the least one compute node in the combination of compute nodes during the task;
• and
• exclude results from the at least one lagging compute node from task results.