Recovering deep learning training

A computer program product comprising:

• one or more computer-readable storage media;
• and
• program instructions stored on the one or more computer-readable storage media to perform operations comprising:
• receiving training data from an external application;
• training a deep learning model at a first training node based on the received trained data;
• saving a plurality of checkpoints in an external memory device during the training of the deep learning model;
• determining that there is an interruption of the training of the deep learning model;
• resuming and completing training of the deep learning model at a second training node in response to determining that there is the interruption of the training of the deep learning model;
• and
• deploying a trained deep learning model in response to completing the training of the deep learning model.

A computer system comprising:

• a processor set;
• one or more computer-readable storage media;
• and
• program instructions stored on the one or more computer-readable storage media to cause the processor set to perform operations comprising:
• receiving training data from an external application;
• training a deep learning model at a first server based on the received trained data;
• saving a plurality of checkpoints in an external memory device during the training of the deep learning model;
• determining that there is an interruption of the training of the deep learning model;
• resuming and completing training of the deep learning model at a second server in response to determining that there is the interruption of the training of the deep learning model;
• and
• deploying a trained deep learning model in response to completing the training of the deep learning model,
• wherein the first server is different from the second server, and the interruption comprises an abnormal failure of the first server.