Smart online link repair and job scheduling in machine learning supercomputers

A method for repairing broken links without interrupting user jobs running on a high-performance network in a Machine Learning (ML) or High-Performance Computing (HPC) supercomputer, the method comprising:

• receiving, by a network controller, information relating to new user jobs coming into the supercomputer;
• performing, by the network controller and a plurality of node agents, a pre-flight check prior to running the user jobs, wherein the pre-flight check includes gathering network state information from a plurality of nodes within the network;
• determining, by the network controller and the plurality of node agents, whether a network error exists in the network based on the network state information;
• monitoring, by the network controller and the plurality of node agents, whether one or more links associated with the network error are in use;
• disconnecting, by the network controller and the plurality of node agents, the one or more links associated with the network error from the rest of the network;
• and
• repairing the disconnected links using a repair workflow initiated by the network controller.