Message aggregation, combining and compression for efficient data communications in GPU-based clusters

A method comprising:

• generating a plurality of network messages;
• wherein in response to an indication that one or more network messages of the plurality of network messages is combinable:
• performing a combining operation on a plurality of data elements in the one or more network messages to produce a given result;
• and
• generating a first packed network message comprising the given result in place of the plurality of data elements, wherein the first packed network message includes fewer than all of the data elements in the one or more network messages;
• replacing two or more network messages with a second packed network message responsive to the two or more network messages comprising an indication of a same destination node and one or more data elements in common, the second packed network message comprising an indication of the destination node and a single copy of the one or more data elements in common.

A computing system comprising:

• a source node;
• a network;
• and
• a target node coupled to the source node via the network;
• and
• wherein the source node is configured to:
• generate a plurality of network messages;
• in response to an indication that one or more network messages of the plurality of network messages is combinable:
• perform a combining operation on a plurality of data elements in the one or more network messages to produce a given result;
• and
• generate a first packed network message comprising the given result in place of the plurality of data elements, wherein the first packed network message includes fewer than all of the data elements in the one or more network messages;
• replace two or more network messages with a second packed network message responsive to the two or more network messages comprising an indication of a same destination node and one or more data elements in common, the second packed network message comprising an indication of the destination node and a single copy of the one or more data elements in common.