US 2003/0118040 A1 · Fibre channel arbitrated loop bufferless switch circuitry to increase bandwidth without significant increase in cost

A bufferless switch for coupling to a plurality of FCAL nets and having a crossbar switch and FCAL loop interface port circuits structured to use the OPN and RRDY primitives of the FCAL protocol for hold back flow control to eliminate the need for a buffer and said ports and crossbar switch structured to provide multiple simultaneous loop tenancies.

A bufferless switch for coupling to a plurality of FCAL nets and having a crossbar switch and FCAL loop interface port circuits structured to use the OPN and RRDY primitives of the FCAL protocol for hold back flow control to eliminate the need for a buffer and said ports and crossbar switch structured to provide multiple simultaneous loop tenancies and dual simplex communication across said switch, each said port including a local bypass data path which can be switched to keep primitives and data frames generated on a local FCAL net coupled to said port routed so as to stay on said local FCAL net such that local loop tenancies on each local FCAL net coupled to a port which do not involve transmission of data or primitives from one port to another can occur simultaneously, and each port including local learning memory means for storing a routing table and scoreboard table, and functioning to fully cache the 8-bit addresses of all nodes on each said local FCAL net coupled to a port along with the ID of the port coupled to each node in said routing table and the busy, available or no privilege status of each other port and whether each busy port is available for dual simplex communications in said scoreboard table, each port structured to write the contents of each said routing table and scoreboard table by learning either from watching local traffic in said port bound for local destination nodes or remote traffic leaving said port and crossing said crossbar switch bound for remote destination nodes or by conducting an active discovery process, each port structured to write the contents of said scoreboard table by learning from watching messages posted on a protocol bus, each said port structured to use destination addresses in OPN primitives transmitted by nodes on said local FCAL net as search keys to search said routing table to determine whether the destination node is local or remote and to which port it is connected and, if local, to switch said local bypass to keep the loop tenancy confined to said local FCAL net, or, if remote, to generate a connection request message to said port coupled to said destination node, each said port being structured to receive such connection requests and respond by arbitrating for control of its local FCAL net and pick a data path through said crossbar switch and generate a reply message naming the data path through said crossbar switch to use in exchanging data frames and primitives, each said port also being structured to respond to receipt or generation of said reply message by generating commands to establish said data path named in said reply message through said crossbar switch, each said port also structured to circulate a fairness token on a fairness token bus coupled to all such ports and to used said fairness token to increase its priority level of access privilege to busy nodes when the fairness token is in possession of the port so that no port can be starved, and then to circulate said fairness token to a neighboring port such that all ports eventually get said fairness token.

A switch for selectively, concurrently coupling the nodes on a plurality of a Fibre Channel Arbitrated Loops together comprising:

a multiplexed bus;
a crossbar switch;
a plurality of switch control means, each for coupling to one local FCAL network, and each coupled to said multiplexed bus and coupled to said crossbar switch, each for receiving OPN primitives from a source node on the local FCAL network to which it is coupled and determining if the destination node identified in said OPN is on said local FCAL network, and, if so, routing the OPN to the destination node via the local FCAL network, but if the destination node is not on said local FCAL, transmitting a destination node location request and the destination address of the destination node identified in said OPN primitive to all said other switch control means via a channel on said multiplexed bus, and for scanning messages received back from said other switch control means via a channel on said multiplexed bus to determine which switch control means and local FCAL network to which flow control primitives and data are to be transferred to accomplish the desired data transfer, and when a grant message is received from another switch control means indicating said other switch control means is the place to send flow control primitives and data and indicating that the FCAL to which said other switch control means is coupled is available for a loop tenancy, for cooperating with said other switch control means to establish a data transfer path across said crossbar switch, and using said data transfer path to transfer flow control primitives between said source and destination nodes, and each switch control means also for receiving destination node location requests and destination addresses, and using said destination address to search a table of destination addresses of nodes on the local FCAL of said switch control means to determine if the destination node identified in said destination node location request is on the local FCAL to which said switch control means is coupled, and, if so, for arbitrating for control of said local FCAL and, when control is won, sending a grant message back to said switch control means which initiated said destination node location request identifying the switch control means coupled to the local FCAL on which said destination node is resident and for sending one or more messages to said switch control means which initiated said destination node location request for purposes of controlling establishment of said data transfer path across said crossbar switch.

A packet switching switch for coupling a plurality of Fibre Channel NL nodes and/or Fibre Channel Arbitrated Loop network (herafter FCAL nets) together to all concurrent data exchanges between a plurality of pairs comprised of one NL node or FCAL net and one other NL node or FCAL net, comprising:

a crossbar switch;
a protocol bus;
a plurality of port circuits each having an input and an output for connecting to an FCAL net coupled to one or more NL nodes and capable of implementing a Fibre Channel loop protocol, and each having a crossbar switch port coupled to said crossbar switch;
means coupled to said protocol bus for maintaining a scoreboard table containing at least status information and a routing table either centrally located or in each port circuit, said routing table containing data mapping destination addresses of NL nodes to port IDs;
and wherein said port circuits are coupled to said means for maintaining a scoreboard table and routing table and wherein said port circuits function to establish connections between themselves by using destination addresses in OPN primitives received from source nodes to search said routing table to determine the ID of a remote port coupled to the destination node having the destination address in the OPN and, using that port ID to search said scoreboard table to determine status of the remote port, and then exchanging messages with said remote port to cause it to arbitrate for and take control of its local FCAL net and establish a particular channel through said crossbar switch and use said channel to transmit primitives and data frames between said source node and said destination node.

2 more elements

A process for communicating data between a source node and a destination node through one or more ports of a switch in a Fibre Channel protocol network, comprising:

1) in a source node, arbitrating for and winning control of a first FCAL net and transmitting an OPN primitive thereon, said OPN primitive having a destination address of a destination node therein;
2) receiving and latching at a first port of said switch said OPN primitive from said source node coupled to said first port by said first FCAL net;
3) using the destination address in said OPN primitive as a search key to search a routing table to find the location of a destination node having said destination address or the ID of a port coupled by an FCAL net to said destination node, or both, and, if the destination node is coupled to said first port, passing said OPN primitive to said destination node via said first port via a local bypass data path coupling an input of the first port to an output of the first port coupled to the first FCAL net, but, if said destination node is coupled to a second port other than said first port, controlling a crossbar switch to establish a data path between said first and second ports and determining if said second port is available, and, if so, sending said OPN primitive to said second port indicating traffic is waiting to be sent to said destination node and latching said OPN in said second port;
4) in said second node, arbitrating for control of a second FCAL net coupled to said second port;
5) when control of said second FCAL net is won following said arbitration, forwarding said OPN to said destination node;
6) receiving an RRDY primitive or a CLS primitive from said destination node in said second port and transmitting said primitive to said source node through said crossbar switch, said first port and said first FCAL net;
and
7) continuing to pass data frames and primitives between said source and destination nodes, until a CLS primitive is transmitted by either said source node or said destination node, and then closing said data path through said crossbar switch and relinquishing control of said first and second FCAL nets.

5 more elements

A packet switching switch for coupling a plurality of Fibre Channel NL nodes and/or Fibre Channel Arbitrated Loop network (herafter FCAL nets) together to all concurrent data exchanges between a plurality of pairs comprised of one NL node or FCAL net and one other NL node or FCAL net, comprising:

a crossbar switch;
a protocol bus;
a plurality of port circuits each having an input and an output for connecting to an FCAL net coupled to one or more NL nodes and capable of implementing a Fibre Channel loop protocol, and each having a crossbar switch port coupled to said crossbar switch;
circuitry in each said port circuit and coupled to said protocol bus for maintaining a routing table containing data mapping destination addresses of NL nodes to port IDs;
and wherein said port circuits function to establish connections between themselves by latching any OPN and RRDY primitives received from source nodes and using destination addresses in OPN primitives received from said source nodes to search said routing table to determine the ID of a remote port coupled to the destination node having the destination address in the OPN, and then establishing a data path through said crossbar switch to said remote port and wherein each said port when it receives an OPN and any following RRDYs from another port via said crossbar switch, functioning to latch said OPN and RRDY and arbitrate for and take control of its local FCAL net, and continuing to try to take conrol of its local FCAL net until control is achieved, and when control is achieved, sending said OPN and any latched RRDYs to the destination node, and thereafter using said data path through said crossbar switch to send primitives and data frames back and forth between said source and destination nodes until the transaction is completed without ever storing any data frames.

2 more elements

In a distributed system comprised of a plurality of nodes which are data consumers and data senders coupled by a switch and a plurality of Fibre Channel Arbitrated Loops, a process for establishing one or more concurrent data transfers between different pairs of nodes comprising the steps:

(1) for each desired data transfer, sending to said switch from a source node a request to start a loop tenancy with a destination node, said request being in the form of a Fibre Channel Arbitrated Loop (hereafter FCAL) primitive and including a tag which identifies the destination node with which data is to be exchanged;
(2) receiving said request and tag in a switching circuit coupled to the FCAL upon which said source node is resident and using said tag to determine from a table of nodes resident on said FCAL upon which said source node is resident if said destination node is resident thereon;
(3) if said destination node is not resident on said FCAL upon which said source node is resident, using a control channel on a multiplexed bus having data transfer channels and control channels to circulate said request and tag to other switching circuits coupled each of said other FCALs;
(4) in each said other switching circuit, using said tag to determine from a table of nodes resident on said FCAL to which said switching circuit is coupled if said destination node is resident on a local FCAL coupled to said switching circuit;
(5) generating in the switching circuit coupled to the FCAL upon which said destination node is resident a grant message identifying the FCAL upon which said destination node is present by inclusion of the identification code of the switching circuit coupled to the FCAL on which said destination node is present in said grant message;
(6) assiging one or more data transfer channels of said multiplexed bus to carry data of said desired data transfer so as to establish a data transfer path between said switching circuit coupled to said FCAL upon which said source node is present and the switching circuit coupled to said FCAL upon which said destination node is present to carry the data and FCAL primitives of the desired FCAL protocol data transfer;
and
(7) using said switching circuits coupled to said FCALs upon which said source and destination nodes are resident to couple an RRDY FCAL protocol primitive from said destination node to said source node after said data transfer path across said multiplexed bus has been established to trigger and control the flow of data, and placing the data of said desired data transfer in one or more of said data transfer channels of said multiplexed bus to carry out said data transfer.

5 more elements

In a distributed system comprised of a plurality of nodes which are data consumers and data senders coupled by a switch and a plurality of Fibre Channel Arbitrated Loops, a process for establishing one or more concurrent data transfers between different pairs of nodes comprising the steps:

(1) for each desired data transfer, sending to said switch from a source node a request to start a loop tenancy with a destination node after arbitrating for and winning control of the FCAL upon which said source node is resident, said request being in the form of FCAL OPN primitive and including a tag which identifies the destination node with which data is to be exchanged;
(2) receiving said request and tag in a switching circuit coupled to the FCAL upon which said source node is resident and using said tag to determine from a table of nodes resident on said FCAL upon which said source node is resident if said destination node is also resident thereon;
(3) if said destination node is not resident on said FCAL upon which said source node is resident, using a control channel on a multiplexed bus having data transfer channels and control channels to broadcast a destination node location request and the destination address of the destination node identified in said OPN primitive, said request requesting other switching circuits coupled each of said other FCALs to search their tables to determine which of them is coupled to the FCAL to which the destination node is connected;
(4) in each said other switching circuit, using said destination address as a search key to search a table of addresses of nodes resident on said FCAL to which said other switching circuit is coupled
(5) if said destination node is resident on a local FCAL coupled to said switching circuit and is available for a transaction, generating a grant message in the switching circuit coupled to the FCAL upon which said destination node is resident, said grant message identifying the FCAL upon which said destination node is present by inclusion of the identification code of the switching circuit coupled to the FCAL on which said destination node is present in said grant message;
(6) establishing a data transfer path between said switching circuit coupled to said FCAL upon which said source node is present and the switching circuit coupled to said FCAL upon which said destination node is present to carry the data and FCAL primitives of the desired FCAL protocol data transfer;
and
(7) using said switching circuits coupled to said FCALs upon which said source and destination nodes are resident and the data path established in step 6 to couple an RRDY FCAL protocol primitive from said destination node to said source node after said data transfer path has been established to trigger and control the flow of data, and transferring data over said data transfer path between said source and destination nodes.

5 more elements

A switch for selectively, concurrently coupling the nodes on a plurality of a Fibre Channel Arbitrated Loops together comprising:

a multiplexed bus;
a plurality of switch control means, each for coupling to one local FCAL network, and each coupled to said multiplexed bus, each for receiving OPN primitives from a source node on the local FCAL network to which it is coupled and determining if the destination node identified in said OPN is on said local FCAL network, and, if so, routing the OPN to the destination node via a local traffic data path in said switch control means and the local FCAL network, but if the destination node is not on said local FCAL, transmitting a destination node location request and the destination address of the destination node identified in said OPN primitive to all said other switch control means via a channel on said multiplexed bus, and for scanning messages received back from said other switch control means to determine which switch control means and local FCAL network to which flow control primitives and data are to be transferred to accomplish the desired data transfer, and when a grant message is received from another switch control means indicating said other switch control means is the place to send flow control primitives and data and indicating that the FCAL to which said other switch control means is coupled is available for a loop tenancy, for cooperating with said other switch control means to establish a data transfer path across said multiplexed bus, and using said data transfer path to transfer flow control primitives between said source and destination nodes, and each switch control means also for receiving destination node location requests and destination addresses, and using said destination address to search a table of destination addresses of nodes on the local FCAL of said switch control means to determine if the destination node identified in said destination node location request is on the local FCAL to which said switch control means is coupled, and, if so, for arbitrating for control of said local FCAL and, when control is won, sending a grant message back to said switch control means which initiated destination node location request identifying the switch control means coupled to the local FCAL on which said destination node is resident and for sending one or more messages to said switch control means which initiated said destination node location request for purposes of controlling establishment of said data transfer path.

A switch for selectively coupling pairs of nodes on a plurality of FCAL nets to each other for one or more simultaneous conversations, comprising:

data path circuitry having a plurality of parallel point-to-point transmit/receive channels and multiplexing circuitry to selectively couple individual ones of said transmit/receive channels to transmit-receive terminals;
a protocol bus;
a plurality of ports each port for interfacing a local FCAL net to said data path circuitry, each port coupled to said protocol bus and comprising:
a serializer/deserializer circuit having terminals for coupling to an FCAL net and having demultiplexing circuitry to convert incoming differential, encoded serial data from said FCAL net to multi-bit characters and having clock recovery circuitry to recover a receive clock therefrom and having multiplexing circuitry for receiving a plurality of multi-bit characters and converting them into a differential, encoded serial data stream;
an elastic buffer to receive multi-bit characters from and send multi-bit characters to said serializer/deserializer circuit and absorb the differences in transmit and receive rate;
a routing table memory storing mappings between destination node addresses and port IDs of the port coupled to each destination node;
scoreboard means for storing status information;
one or more state machine means coupled to said elastic buffer, said routing table memory, said scoreboard means and said protocol bus and at least to said transmit-receive terminals of said data path circuitry for using said routing table memory contents for determining whether destination nodes identified in OPN primitives received from source nodes on said local FCAL net are local or remote and using the data content of said scoreboard means to determine if connection to-a destination port coupled to said destination node is permissible and transmitting a connection request on said protocol bus to said destination port requesting a connection thereto and using the contents of a connect response frame output by said state machine means of said destination port on said protocol bus for controlling said data path circuitry to implement full duplex connections between said local source and remote destination nodes or to implement full or half duplex connections to remote destination nodes, and for implementing logic and control functions to control said data path circuitry to carry out a dual simplex connections so as to send data and primitives from a source node on said local FCAL net to a remote dual simplex destination node coupled to some other port which is a source node in another half duplex loop tenancy and receive buffer credit flow control primitives back from said dual simplex destination node and to respond to each said buffer credit flow control primitive by transmitting a frame of data to said dual simplex destination node, said state machine means also structured for sending flow control primitives to a third port for transmission to a third node and receive OPN and CLS primitives and data frames from said third node when said third port has established a dual simplex connection to said port through said data path circuitry, and transmit data frames and CLS primitives received from said third node to said source node if and only if said data frames are directed to said source node as indicated by said OPN received from said third node.

5 more elements

A process for carrying out dual simplex communications in a FCAL switch coupling a plurality of FCAL nets, comprising:

receiving and latching an OPN and at least the source node address thereof identifying the address of said source node and receiving any following buffer credit RRDY primitives and either storing said RRDYs or counting the number thereof at a source port from a source node on the local FCAL net coupled to said source port, said OPN directed to a remote destination node on the FCAL net coupled to a remote port;
establishing a front channel connection through a backplane between said source port and said destination port;
converting said OPN to half duplex and transmitting said half duplex OPN to said destination OPN over said front channel;
receiving any RRDY primitives from said destination node and transmitting them to said source node via a back channel;
receiving a connection request message from a third port coupled to a third node and granting said connection request by a connection grant message to said third port coupled to said third node, said connection grant message naming a backplane channel to use for a back channel connection between said source node and said third node;
establishing a backchannel connection between said source port and said third port through said backplane channel named in said connection grant message;
transmitting an OPN directed to said source node from said third node to said source port;
comparing the destination node address in said OPN from said third port to the node address of the OPN originally received by said source port, and, if the addresses match, quashing the OPN from said third port and either transmitting one stored RRDY received from said source node to said third node or transmitting a message which causes said third port to generate an RRDY and transmit it to said third node;
transmitting one frame of data from said third node to said source node via said back channel connection;
and
continuing to intercept RRDY primitives transmitted by said source node and send them to said third node and to receive frames of data transmitted in response to receipt at said third node of said RRDYs and transmit said frames to said source node via said back channel connection and mix said frames of data on said back channel connection with RRDY primitives received from said destination node.

8 more elements

A process for dual simplex communication using a switch coupling a plurality of FCAL nets, comprising:

1) receiving a full duplex OPN (Dest, Src) from a source node designating a destination node address, Dest, and a source node address, Src, and converting the full duplex format OPN (Dest, Src) to a simplex or half duplex format OPN (Dest, Dest) and establishing a front channel connection to said destination node;
2) storing the full duplex OPN including at least the source node address in a source port of said switch coupled to said source node;
3) stripping any buffer credit RRDYs output by said source node and not transmitting said RRDYs output by said source node to said destination node of said front channel connection, but either storing or counting said RRDYs output by said source node and making them available to a third node to send data to said source node either by transmitting stored RRDYs output by said source node one at a time to said third node or by conveying the number of RRDYs output by said source node to a third port coupled to said third node and synthesizing in said third node a number of RRDYs equal to the value of said count and sending said RRDYs to said third node one at a time;
4) updating the status data in a scoreboard memory of each port to indicate which busy ports are coupled to source nodes of front channel connections and thus are available to receive frames in dual simplex communication from third nodes;
5) establishing a back channel connection from said third node to said source node of said front channel connection and transmitting an OPN from said third node to said source port, and using said stored source node address from said OPN transmitted by said source node and stored in said source port as an access key to compare to the destination node address of said OPN transmitted to said source port by said third node via said back channel connection and allowing dual simplex communication to proceed on said back channel if said source node address matches said destination node address of said OPN from said third node, but if there is no match, blocking dual simplex communication with this particular third node and closing said back channel connection;
6) deleting said OPN from said third node in said source port and never transmitting it to said source node if the destination address of the OPN from said third node matches the stored source address of the stored OPN from said source node;
and
7) receiving data frames transmitted by said third node at said source port with one data frame transmitted in response to each RRDY received by said third node, and mixing said data frames in with RRDYs received from said destination node via said front channel connection and transmitting the combined data frames and RRDYs to said source node via said back channel connection.

5 more elements

A switch apparatus comprising a plurality of integrated circuits each comprising a slice of an FCAL switch and coupled together to form said FCAL switch, each integrated circuit having a plurality of port circuits, each port circuit having substantially less buffer memory to store data than prior art FL ports in prior art fabric switches, each said integrated circuit also comprising a portion of a distributed, scaleable crossbar switch such that any number of said integrated circuit slices can be coupled together to form an FCAL switch which has as many ports as are necessary for the size of network in which the switch is to be used, limited only by the available FCAL address space of 128 nodes.

An FCAL buffered switch for coupling to a plurality of FCAL nets and having a crossbar switch and FCAL loop interface port circuits structured to use the destination address of the OPN to find the destination node and destination port and including switching circuitry coupled to said crossbar switch by by multiple paths including at least a straight through path and paths through one or more buffer memorys to exchange data frames and FCAL primitives with one or more destination nodes and structured to use one or more buffer memories to store data output by a source node destined for a destination node which has a busy status at the time the data is output by said source node, said ports and crossbar switch structured to provide multiple simultaneous loop tenancies and said ports and crossbar switch also structured to provide dual simplex communication across said switch, each said port including a local bypass data path which can be switched to keep primitives and data frames generated on a local FCAL net coupled to said port routed so as to stay on said local FCAL net such that local loop tenancies on each local FCAL net coupled to a port which do not involve transmission of data or primitives from one port to another can occur simultaneously, and each port including local learning memory means for storing a routing table and scoreboard table, and functioning to fully cache the 8-bit addresses of all nodes on each said local FCAL net coupled to a port along with the ID of the port coupled to each node in said routing table and the busy, available or no privilege status of each other port and whether each busy port is available for dual simplex communications in said scoreboard table, each port structured to write the contents of each said routing table and scoreboard table by learning either from watching local traffic in said port bound for local destination nodes or remote traffic leaving said port and crossing said crossbar switch bound for remote destination nodes or by conducting an active discovery process, each port structured to write the contents of said scoreboard table by learning from watching messages posted on a protocol bus, each said port structured to use destination addresses in OPN primitives transmitted by nodes on said local FCAL net as search keys to search said routing table to determine whether the destination node is local or remote and to which port it is connected and, if local, to switch said local bypass to keep the loop tenancy confined to said local FCAL net, or, if remote, to generate a connection request message to said port coupled to said destination node, each said port being structured to receive such connection requests and respond by arbitrating for control of its local FCAL net and pick a data path through said crossbar switch and generate a reply message naming the data path through said crossbar switch to use in exchanging data frames and primitives, each said port also being structured to respond to receipt or generation of said reply message by generating commands to establish said data path named in said reply message through said crossbar switch, each said port also structured to circulate a fairness token on a fairness token bus coupled to all such ports and to used said fairness token to increase its priority level of access privilege to busy nodes when the fairness token is in possession of the port so that no port can be starved, and then to circulate said fairness token to a neighboring port such that all ports eventually get said fairness token.

Fibre channel arbitrated loop bufferless switch circuitry to increase bandwidth without significant increase in cost

Life of this patent

Fibre channel arbitrated loop bufferless switch circuitry to increase bandwidth without significant increase in cost

Life of this patent

Abstract

Claims52