System and method for preserving message order while employing both programmed I/O and DMA operations

A method of transmitting messages between nodes of a distributed computer system using a combination of memory mapped programmed I/O (PIO) commands and memory mapped DMA commands to transmit each of a subset of the messages, comprising steps of:

• establishing a plurality of queues in the first node's network interface;
• initiating the transmission of a multipart message from a first node to a second node of the distributed computer system by sending to a network interface of the first node a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of the multipart message to be stored in a respective specified memory mapped location in the second node, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;
• assigning each of the PIO store and DMA store commands that specify a respective component of the multipart message to a same one of the queues;
• at the first node's network interface:
• receiving the sequence of PIO and DMA commands corresponding to the multipart message;
• storing each PIO and DMA command received by the first node's network interface in the assigned one of the queues, wherein the PIO and DMA commands stored in each queue are stored in a same order as they are received by the first node's network interface, which corresponds to the predefined message component order;
• for each queue, packetizing the PIO and DMA commands therein so as to generate an ordered stream of data transfer packets whose order corresponds to the order in which the PIO and DMA commands are stored in the queue, and transmitting each of the ordered stream of data transfer packets to a destination node specified therein, the transmitting step preserving packet order within the ordered stream of data transfer packets, at least with respect to packets being transmitted to a same destination node, so as to store the respective components of the multipart message in their respective specified memory locations in the destination node in the predefined message component order.

A network interface for a first node of a distributed computer system, comprising:

• receive logic for receiving a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of a multipart message to be stored in a respective specified memory mapped location in a second node of the distributed computer system, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;
• the receive logic including a queue for storing each of the received PIO and DMA commands, wherein the PIO and DMA commands are stored in the queue in a same order as they are received, which corresponds to the predefined message component order;
• means, coupled to the receive logic, for packetizing the PIO and DMA commands stored in the queue to generate an ordered stream of data transfer packets whose order corresponds to the predefined message component order;
• and
• means, coupled to the packetizing means, for transmitting the ordered stream of data transfer packets to the second node so as to store the respective components of the multipart message in their respective specified memory locations in the second node in a same order as the predefined message component order;
• the transmitting means preserving packet order within the ordered stream of data transfer packets.

A network interface for a first node of a distributed computer system, comprising:

• receive logic for receiving a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of a multipart message to be stored in a respective specified memory mapped location in a second node of the distributed computer system, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;
• the packetizing means including:
• a plurality of queues;
• classification logic for assigning each of the PIO store and DMA store commands that specify a respective component of the multipart message to a same one of the queues and for storing each PIO and DMA command received by the first node's network interface in the assigned one of the queues, wherein the PIO and DMA commands stored in each queue are stored in a same order as they are received by the first node's network interface, which corresponds to the predefined message component order;
• and
• packetizing logic, for each queue, for packetizing the PIO and DMA commands therein so as to generate an ordered stream of data transfer packets whose order corresponds to the order in which the PIO and DMA commands are stored in the queue;
• the transmitting means including means for transmitting each of the ordered stream of data transfer packets to a destination node specified therein while preserving packet order within the ordered stream of data transfer packets at least with respect to packets being transmitted to a same destination node.

A network interface for a first node of a distributed computer system, comprising:

• receive logic for receiving a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of a multipart message to be stored in a respective specified memory mapped location in a second node of the distributed computer system, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;
• the receive logic including a queue for storing each of the received PIO and DMA commands, wherein the PIO and DMA commands are stored in the queue in a same order as they are received, which corresponds to the predefined message component order;
• command processing circuitry, coupled to the receive logic, for packetizing the sequence of PIO and DMA commands stored in the queue to generate an ordered stream of data transfer packets whose order corresponds to the predefined message component order;
• and
• packet handling circuitry, coupled to the command processing circuit, for transmitting the ordered stream of data transfer packets to the second node so as to store the respective components of the multipart message in their respective specified memory locations in the second node in the predefined message component order;
• the packet handling circuitry preserving packet order within the transmitted ordered stream of data transfer packets.

A network interface for a first node of a distributed computer system, comprising:

• receive logic for receiving a sequence of PIO store and DMA store commands, each PIO store and DMA store command specifying a respective component of a multipart message to be stored in a respective specified memory mapped location in a second node of the distributed computer system, the sequence of the PIO store and DMA store commands corresponding to a predefined message component order;
• command processing circuitry, coupled to the receive logic, including:
• a plurality of queues;
• classification logic for assigning each of the PIO store and DMA store commands that specify a respective component of the multipart message to a same one of the queues and for storing each PIO and DMA command received by the first node's network interface in the assigned one of the queues, wherein the PIO and DMA commands stored in each queue are stored in a same order as they are received by the first node's network interface, which corresponds to the predefined message component order;
• and
• a packetizing engine, for each queue, for packetizing the PIO and DMA commands therein so as to generate an ordered stream of data transfer packets whose order corresponds to the order in which the PIO and DMA commands are stored in the queue;
• and
• packet handling circuitry, coupled to the each of the queues in the command processing circuitry, including a scheduler for scheduling transmission of the ordered stream of data transfer packets in each queue to the destination nodes specified therein while preserving packet order within each queue's ordered stream of data transfer packets, at least with respect to packets being transmitted to a same destination node, so as to store the respective components of the multipart message in their respective specified memory locations in the destination node in the predefined message component order.