System and method of maintaining coherency in a distributed communication system

A method of maintaining order of transactions issued in a distributed communication system, the distributed communication system comprising a plurality of nodes interconnected by a plurality of communication links, the plurality of nodes configured to access a plurality of addressable memory locations for storing data, the plurality of nodes comprising a source node and a target node, the method comprising the acts of:

• storing at the target node a first request received from the source node and directed to a first memory address;
• storing at the target node a second request directed to the first memory address;
• servicing the first request;
• transmitting a communication to the source node in response to the act of servicing the first request;
• stalling an act of servicing the second request pending receipt by the target node of a source response transmitted from the source node in response to the communication;
• wherein the acts of storing the first request and the second request at the target node comprises the act of ordering the first request and the second request in a queue in the order in which the first request and the second request were received;
• wherein the first request comprises a read request, and wherein the communication transmitted to the source node comprises a plurality of responses issued from the plurality of nodes;
• wherein the plurality of responses comprises a memory access response issued from the target node, and wherein the act of servicing the first request comprises the acts of:
• accessing, by the target node, a memory location associated with the first memory address;
• transmitting to the source node the memory access response based on the act of accessing by the target node;
• and
• determining whether data corresponding to the first memory address is cached at any of the plurality of nodes;
• wherein the plurality of responses comprises a plurality of cache responses issued by the plurality of nodes, and wherein the act of determining whether data corresponding to the first memory address is cached at any of the plurality of nodes comprises the acts of:
• issuing a probe directed to each of the plurality of nodes;
• and
• issuing, by each of the plurality of nodes, one of the cache responses in response to the probe, each of the cache responses being directed to the source node;
• and
• wherein if a first node of the plurality nodes is caching data corresponding to the first memory address, the method comprises the acts of:
• issuing, by the first node, a memory cancel response directed to the target node;
• and
• formatting the cache response issued from the first node to indicate the act of issuing the memory cancel response.

A communication node for a distributed communication system comprising a plurality of communication nodes interconnected by a plurality of communication links, the node comprising:

• a memory controller to control access to a memory, the memory comprising a plurality of memory locations corresponding to a plurality of memory addresses;
• an interface configured to connect to a communication link;
• and
• communication logic coupled to the memory controller, and the interface, wherein the communication logic is configured to:
• store a first request received from a source via the interface, the first communication being directed to a first memory address of the plurality of memory addresses;
• store a second communication directed to the first memory address;
• generate a first response directed to the source in response to the first request;
• and
• stall the second request pending receipt from the source of a second response in response to the first response;
• wherein the communication logic is configured to generate a probe for transmission to each of the plurality of communication nodes in the distributed communication system, the probe to determine whether data corresponding to the first memory address is cached at any of the plurality of communication nodes;
• wherein the first request comprises a read request, and wherein, the communication logic is configured to issue the read request to the memory controller to access the memory location corresponding to the first memory address;
• wherein the communication logic is configured to cancel the access by the memory controller to the memory location in response to a memory cancel response received from a particular communication node of the plurality of communication nodes, the memory cancel response indicating that the particular communication node is caching data corresponding to the first memory address.

A distributed communication system, comprising:

• a plurality of nodes, the plurality of nodes comprising a source node and a target node;
• a plurality of communication links interconnecting the plurality of nodes;
• and
• a memory accessible by the plurality of nodes, the memory comprising a plurality of memory locations corresponding to a plurality of memory addresses, wherein each of the plurality of nodes is configured to control access to a portion of the memory locations;
• wherein the source node is configured to dispatch a first request directed to a first memory address accessible by the target node;
• wherein the target node is configured to transmit a first response to the source node in response to the first request;
• wherein the source node is configured to transmit a second response to the target node in response to the first response;
• and
• wherein the target node is configured to stall service of a second request directed to the first memory address pending receipt of the second response;
• wherein each of the plurality of nodes comprises a cache to store data, and wherein the target node is configured to issue a probe in response to the first request, the target node directing the probe to each of the plurality of nodes to determine whether data corresponding to the first memory address is stored in the cache of any of the plurality of nodes, and wherein each of the plurality of nodes is configured to issue a cache response in response to the probe, the cache response indicating whether data is stored in the cache of the respective node;
• wherein the first request comprises a read request, and wherein a first node of the plurality of nodes is configured to issue a memory cancel response directed to the target node if the first node is storing data corresponding to the first memory address in its cache;
• wherein the first node is configured to format the respective cache response to indicate that the first node has issued the memory cancel response;
• wherein the source node is configured to format the second response to indicate issuance of the memory cancel response based on the cache response;
• and
• wherein, in response to the second response, the target node is configured to stall the second request pending receipt of the memory cancel response.

A method of maintaining order of transactions in a distributed communication system, the distributed communication system comprising a plurality of nodes interconnected by a plurality of communication links, the plurality of nodes having access to a plurality of addressable memory locations, the plurality of nodes comprising a source node and a target node, the method comprising the acts of:

• dispatching, by the source node, a first request directed to a first memory address accessible by the target node;
• transmitting, from the target node, a first response directed to the source node in response to the first request;
• transmitting, from the source node, a second response directed to the target node after receipt of the first response;
• and
• stalling service, by the target node, of a second request directed to the first memory address pending receipt of the second response;
• issuing, by the target node, a probe in response to the first request, the probe being directed to each of the plurality of nodes to determine whether any of the plurality of nodes is caching data corresponding with the first memory address;
• and
• issuing, by each of the plurality of nodes, a third response in response to the probe,
• wherein act of transmitting the second response, from the source node, is performed after receipt of all of the third responses;
• wherein the first request comprises a read request, and wherein if the probe determines that a first node of the plurality of nodes is caching data corresponding to the first address, then the method comprises the act of:
• issuing, by the first node, a memory cancel response directed to the target node to cancel a memory access by the target node to the first memory address.