# US 2022/0045969 A1 · Mapping nvme-over-fabric packets using virtual output queues > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2022/0045969 A1 - Document ID: US20220045969A1 - Status: Published application - Kind: A1 - Application number: 17/413,997 - Filed: 2019-06-25 - Published: 2022-02-10 - Earliest claimed: 2019-06-25 - Assignees: Hewlett Packard Enterprise Development LP - Applicants: Hewlett Packard Enterprise Development LP - Inventors: Brian Peter L'Ecuyer; Charles Tuffli; Giuseppe Scaglione - Web page: https://patentgrep.com/patent/US20220045969A1 ## CPC - G06F 3/0659: {Command handling arrangements, e.g. command buffers, queues, command scheduling} - G06F 3/061: {Improving I/O performance} - G06F 3/0679: {Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]} - H04L 49/109: Integrated on microchip, e.g. switch-on-chip - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract A network infrastructure device (e.g., network switch), that integrates solid-state drive (SSD) storage, using Non-volatile Memory Express (NVMe) data transfer protocol, for use by remote application hosts is provided. High availability configurations of network switches using direct rate control (RC) feedback for a plurality of submission queues mapped to SSD storage is provided. NVMe over fabric (NVMe-oF) is an implementation of NVMe protocol over a network fabric. Access to SSDs over network fabrics may be controlled using a direct RC feedback signal between an egress queue congestion accounting (associated with a single egress output) and a source node receiving input/output commands from remote hosts for the integrated SSD devices. In some implementations, direct RC feedback signals use hardware based signals. In some implementations, direct RC feedback signals are implemented in the hardware logic (silicon chip logic) within an internal switch fabric of the network switch. ## Claims ### Claim 1 A method comprising: receiving a first network transmission at a network infrastructure device provided from a remote application communicatively coupled via a network to the network infrastructure device, the first network transmission associated with a first data transfer operation using Non Volatile Memory Express over fabric (NVMe-oF) protocol; enqueueing the first network transmission to one or more submission commands to provide to an internal switch fabric of the network infrastructure device, the one or more submission commands associated with a plurality of respective submission queues of an NVMe storage driver interfacing with the internal switch fabric and an NVMe storage device, wherein the plurality of respective submission queues are mapped to at least a first virtual output queue and a second virtual output queue of the NVMe storage driver, the first virtual output queue being mapped to a first completion queue of the NVMe storage driver, and the second virtual output queue being mapped to a second completion queue of the NVMe storage driver; receiving a second network transmission at the network infrastructure device, the second network transmission associated with a second data transfer operation using NVMe-oF protocol; and controlling a flow of the second network transmission to be enqueued and processed using a direct rate control (RC) feedback signal between an egress queue congestion accounting and a source node receiving at least one of the first network transmission or the second network transmission. ### Claim 2 (depends on 1) The method of claim 1, wherein the direct RC feedback signal is a hardware based signal. ### Claim 3 (depends on 2) The method of claim 2, wherein the hardware based signal is implemented using hardware logic implemented within the internal switch fabric. ### Claim 4 (depends on 1) The method of claim 1, wherein the network infrastructure device is a network switch. ### Claim 5 (depends on 1) The method of claim 1, wherein the network infrastructure device is a high availability network switch. ### Claim 6 (depends on 5) The method of claim 5, wherein the high availability network switch provides high availability access to a plurality of integrated solid state drives including the NVMe storage device. ### Claim 7 (depends on 1) The method of claim 1, wherein the first network transmission is received at a first port of the network infrastructure device and the second network transmission is received at a second port of the network infrastructure device different from the first port. ### Claim 8 A network switch comprising: a first input port to receive a first plurality of submission commands from a first remote host, the first plurality of submission commands based on Non Volatile Memory Express (NVMe) protocol; a first plurality of submission queues in which the first plurality of submission commands are enqueued, the first plurality of submission queues mapped, via an internal switch fabric, to a single egress output to provide data transfer operations with an integrated NVMe based storage device; a first completion queue and a second completion queue each associated with a respective one of the first plurality of submission queues, wherein a plurality of response commands are enqueued for transmission using NVMe protocol to the first remote host; and a NVMe storage driver to interface the first plurality of submission commands from the first plurality of submission queues via the internal switch fabric to the integrated NVMe based storage device, wherein control flow for the first plurality of submission queues includes a direct rate control (RC) feedback between the single egress output and the first plurality of submission queues. ### Claim 9 (depends on 8) The network switch of claim 8, wherein the NVMe storage driver is to control the flow of submission commands by determining whether at least one of the first completion queue and the second completion queue is available based on a signal from the direct RC feedback. ### Claim 10 (depends on 9) The network switch of claim 9, wherein the NVMe storage driver is to control the flow of submission commands by: obtaining a submission command from a first virtual output queue responsive to determining that the first completion queue is available; or obtaining the submission command from a second virtual output queue responsive to determining that the second completion queue is available. ### Claim 11 (depends on 10) The network switch of claim 10, wherein the NVMe storage driver is to: dequeue the submission command obtained from the first virtual output queue or the second virtual output queue; and fulfill the submission command by performing a data transfer operation with the integrated NVMe based storage device. ### Claim 12 (depends on 11) The network switch of claim 11, wherein the NVMe storage driver is to: enqueue a response command responsive to the submission command in the first completion queue responsive to determining that the submission command that was obtained from the first virtual output queue has been processed; or enqueue the response command in the second completion queue responsive to determining that the submission command that was obtained from the second virtual output queue has been processed. ### Claim 13 (depends on 9) The network switch of claim 9, wherein: information from the first completion queue or the second completion queue is provided via a response network message using NVMe protocol to the remote host. ### Claim 14 (depends on 8) The network switch of claim 8, further comprising: a second input port to receive a second plurality of submission commands from a second remote NVMe host; a third output port to transmit response commands to the second remote NVMe host; a second plurality of submission queues in which the second plurality of submission commands received via the second input port are enqueued; a third virtual output queue associated with the second plurality of submission queues mapped to the first completion queue; and a fourth virtual output queue associated with the second plurality of submission queues mapped to the second completion queue, wherein the NVMe storage driver is to control the flow, of the submission commands using egress queue congestion accounting and a direct RC feedback associated with respective ones of the first completion queue and the second completion queue. ### Claim 15 (depends on 8) The network switch of claim 8, further comprising: a high availability (HA) configuration of a plurality of integrated NVMe based storage devices wherein pairs of the plurality of integrated NVMe based storage devices provide redundancy for a respective member of each pair. ### Claim 16 (depends on 15) The network switch of claim 15, wherein at least one pair of the pairs of the plurality of integrated NVMe based storage devices includes a first SSD on a first line card of the network switch and a second SSD on a second line card of the network switch, the first line card plugged into a different card slot of the network switch than the second line card. ### Claim 18 A non-transitory computer readable medium comprising computer executable instructions stored thereon that, when executed by a processor in a network switch, cause the processor to: receive a first network transmission at a network infrastructure device provided from a remote application communicatively coupled via a network to the network infrastructure device, the first network transmission associated with a data transfer operation using Non Volatile Memory Express over fabric (NVMe-oF) protocol; enqueue the first network transmission to one or more first submission commands to provide to an internal switch fabric of the network infrastructure device, the one or more first submission commands associated with a plurality of respective submission queues of an NVMe storage driver interfacing with the internal switch fabric and an NVMe storage device, wherein the plurality of respective submission queues are included in a first virtual output queue and a second virtual output queue of the NVMe storage driver, the first virtual output queue being mapped to a first completion queue of the NVMe storage driver, and the second virtual output queue being mapped to a second completion queue of the NVMe storage driver; receive a second network transmission at the network infrastructure device, the second network transmission associated with a data transfer operation using NVMe-oF protocol; and control a flow of the second network transmission to be enqueued to the one or more submission commands using a direct rate control (RC) feedback signal between an egress queue congestion accounting and a source node receiving at least one of the first network transmission or the second network transmission. ### Claim 19 (depends on 18) The non-transitory computer readable medium of claim 18, wherein the direct RC feedback signal is a hardware based signal. ### Claim 20 (depends on 19) The non-transitory computer readable medium of claim 19, wherein the hardware based signal is implemented using hardware logic implemented within an internal switch fabric of the network switch.