Memory-efficient neural network training

One or more non-transitory computer-readable media comprising instructions that, upon execution of the instructions by a first worker of a distributed neural network (NN) training system, are to cause the first worker to:

• execute a forward training pass of a first node of a distributed NN, wherein execution of the forward training pass includes generation of a first computational graph (CG) that is based on inputs related to a second node that is processed by a second worker of the distributed NN;
• delete, subsequent to the forward training pass of the first node, the CG;
• and
• execute, a backward pass of the first node, wherein execution of the backward pass includes re-generation of at least a portion of the first CG.

A method comprising:

• executing, by a first worker of a distributed neural network (NN), a forward training pass of a first node of the distributed NN, wherein execution of the forward training pass includes generation of a first computational graph (CG) that is based on inputs related to a second node that is processed by a second worker of the distributed NN;
• deleting, by the first worker subsequent to the forward training pass of the first node, the CG;
• and
• executing, by the first worker, a backward pass of the first node, wherein execution of the backward pass includes re-generation of at least a portion of the first CG.