Prioritize the earlier step messages for collective algorithms

A network device comprising:

• a circuitry configured to:
• receive a first packet and a second packet from a node of a collective algorithm, wherein the first packet corresponds to a first hierarchical level of the collective algorithm, and the second packet corresponds to a second hierarchical level of the collective algorithm;
• allocate a first amount of bandwidth to the first packet and a second, different amount of bandwidth to the second packet;
• and
• transmit, in parallel, the first packet to a first destination node using the first amount of bandwidth, and the second packet to a second destination node using the second, different amount of bandwidth.

A system comprising:

• a node, wherein the node is configured to:
• generate a first packet, and a second packet, wherein the first packet comprises an indication of an association with a first hierarchical level of a collective algorithm, and the second packet comprises an indication of an association with a second hierarchical level of the collective algorithm;
• and
• a network device, wherein the network device is configured to:
• receive, from the node, the first packet and the second packet;
• allocate a first amount of bandwidth to the first packet and a second, different amount of bandwidth to the second packet;
• and
• transmit, in parallel, the first packet to a first destination node using the first amount of bandwidth, and the second packet to a second destination node using the second, different amount of bandwidth.