Cluster-based training methods, devices, electronic equipment and storage media

An electronic device, comprising:

• at least one processor;
• and
• a memory communicatively connected to the at least one processor;
• wherein the memory stores instructions executable by the at least one processor, and
• when the instructions are executed by the at least one processor, the at least one processor is caused to implement a cluster-based training method, wherein the cluster comprises a training node for executing a model training task and a plurality of standby nodes, and the method comprises:
• in response to a hardware fault in the training node, selecting a target standby node from the plurality of standby nodes, and obtaining a target training snapshot of the model training task in the training node, wherein the target training snapshot comprises training state data of the model training task;
• and
• initializing the target standby node based on a container image of a model training program in the training node and the training state data to replace the training node with the target standby node to continue executing the model training task.