A non-transitory machine readable medium storing a program of forwarding data messages between graphics processing units (GPUs) that are connected through a network and that perform computations to collectively execute a distributed application, the program comprising sets of instructions for:
- specifying, for each computation of each source GPU, a data message flow comprising one or more data messages that store a result of the computation, said generating comprising specifying for each data message a header comprising (i) a tag that is not a network address and that is used for forwarding the data message through the network and (ii) one or more fields storing one or more transport layer protocol (TLP) parameters;
- using the TLP parameters as layer 4 (L4) parameters to establish forwarding of each data message flow between the source EPU and the destination for the flow.