System and method of drainless link repair for increased accelerator utilization during failure

A method for troubleshooting an accelerator architecture comprising a plurality of accelerator hosts, the method comprising:

• automatically detecting a failure in an interconnect connecting two accelerator elements in the accelerator architecture in a health monitor service executed by a processor in each of the plurality of accelerator hosts;
• in the link health monitor service, identifying a first accelerator element and a second accelerator element connected by the failed interconnect;
• and
• identifying in a pod manager, a slice of accelerator elements, the slice including both the first accelerator element and the second accelerator element, the slice being organized in a hierarchical tree, and disabling accelerator elements in the slice of accelerator elements and all parent nodes of the identified slice of accelerator elements;
• and
• in a scheduler, sending accelerator tasks to an accelerator element in a different slice than the slice including both the first and second accelerator elements.

A non-transitory computer readable medium storing instructions executable by a computer processor, that when executed by the computer processor, cause the computer processor to:

• in an accelerator architecture comprising a plurality of accelerator components connected to each other through high-speed interconnects, detecting a failure of at least one of the interconnects;
• communicating the detected failure of the interconnect to a pod manager;
• in the pod manager, determining a slice defining a group of connected accelerator components containing a first accelerator component and a second accelerator component connected by the failed interconnect, the slices arranged in a hierarchical tree;
• marking the determined slice and all parent slices of the determined slice in the hierarchical tree as affected by the failed interconnect;
• and
• identifying a scheduler of the affected slices, wherein the scheduler does not schedule any processing tasks to the affected slices.