Topology aware communications at the collective transport layer

A non-transitory computer readable storage medium storing instructions that when executed by one or more computer processors cause the one or more computer processors to perform steps comprising:

• communicating messages between a sender host and a receiver host via a collective transport layer, each host comprising a plurality of processing devices and a Remote Direct Memory Access (RDMA) Network Interface Card (R-NIC);
• detecting network congestion between the sender host and the receiver host;
• in response to detecting network congestion, selecting a proxy processing device based on a topological distance of the proxy processing device from a target R-NIC of the sender host;
• scattering, from a source processing device to the proxy processing device, message data through a high-bandwidth intra-node interface;
• transmitting, from the proxy processing device, the message data via the target R-NIC of the sender host to the receiver host;
• and
• receiving, at the receiver host, the message data at a destination processing device of the receiver host via a destination R-NIC of the receiver host.

A computer system comprising:

• one or more computer processors;
• and
• a non-transitory computer readable storage medium storing instructions that when executed by one or more computer processors cause the one or more computer processors to perform steps comprising:
• communicating messages between a sender host and a receiver host via a collective transport layer, each host comprising a plurality of processing devices and a Remote Direct Memory Access (RDMA) Network Interface Card (R-NIC);
• detecting network congestion between the sender host and the receiver host;
• in response to detecting network congestion, selecting a proxy processing device based on a topological distance of the proxy processing device from a target R-NIC of the sender host;
• scattering, from a source processing device to the proxy processing device, message data through a high-bandwidth intra-node interface;
• transmitting, from the proxy processing device, the message data via the target R-NIC of the sender host to the receiver host;
• and
• receiving, at the receiver host, the message data at a destination processing device of the receiver host via a destination R-NIC of the receiver host.