# US 12,010,173 B2 · Class-based queueing for scalable multi-tenant RDMA traffic > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,010,173 B2 - Document ID: US12010173B2 - Status: Granted - Kind: B2 - Application number: 17/166,922 - Filed: 2021-02-03 - Published: 2022-06-30 - Granted: 2024-06-11 - Earliest claimed: 2020-12-30 - Assignees: Oracle International Corporation - Applicants: Oracle International Corporation - Inventors: Santosh Narayan Shilimkar; David Dale Becker; Jagwinder Singh Brar - Web page: https://patentgrep.com/patent/US12010173B2 ## CPC - H04L 67/1097: for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS] - H04L 12/4633: {Interconnection of networks using encapsulation techniques, e.g. tunneling} - H04L 12/4641: {Virtual LANs, VLANs, e.g. virtual private networks [VPN] (LAN interconnection over a bridge based backbone H04L12/462; encapsulation techniques H04L12/4633; routing of packets H04L45/00; packet switches H04L49/00; virtual private networks for security H04L63/0272)} - H04L 12/4645: {Details on frame tagging (routing of packets H04L45/00; support for virtual LAN H04L49/354)} - H04L 47/115: {using a dedicated packet} - H04L 47/2408: for supporting different services, e.g. a differentiated services [DiffServ] type of service - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04L 67/55: Push-based network services - H04L 67/562: Brokering proxy services - H04L 67/565: Conversion or adaptation of application format or content (adding application control or application functional data H04L67/561) - H04L 67/61: taking into account QoS or priority requirements - H04L 69/22: Parsing or analysis of headers - H04L 69/325: in the network layer [OSI layer 3], e.g. X.25 (H04L69/16 takes precedence) - H04L 69/40: for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection (management of faults, events, alarms or notifications in data switching networks H04L41/06) - H04L 2212/00: Encapsulation of packets - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Techniques and apparatus for data networking are described. In one example, a method of queuing Remote Direct Memory Access (RDMA) packets includes receiving a first RDMA packet having a first quality-of-service (QoS) data field; based on a value of the first QoS data field, queueing the first RDMA packet in a first queue of a plurality of queues; receiving a second RDMA packet having a second QoS data field; and based on a value of the second QoS data field, queueing the second RDMA packet in a second queue of the plurality of the queues, the second queue being different than the first queue. ## Claims ### Claim 1 A method of queuing Remote Direct Memory Access (RDMA) packets, the method comprising: by a networking device, receiving a first RDMA packet having a first quality-of-service (QoS) data field and a first virtual local area network (VLAN) identifier, the first VLAN identifier indicating a first VLAN; based on a quality-of-service value of the first QoS data field that indicates a class of service for the first RDMA packet, queueing the first RDMA packet in a first RDMA queue of a plurality of queues; by the networking device, receiving a second RDMA packet having a second QoS data field and a second VLAN identifier, the second VLAN identifier indicating a second VLAN; based on a quality-of-service value of the second QoS data field that indicates a class of service for the second RDMA packet, queueing the second RDMA packet in a second RDMA queue of the plurality of the queues, the second RDMA queue being different than the first RDMA queue; by the networking device, receiving a third RDMA packet having a third QoS data field and a third VLAN identifier, the third VLAN identifier indicating a third VLAN that is different than the first VLAN; and based on a quality-of-service value of the third QoS data field that indicates a class of service for the third RDMA packet being the same as the quality-of-service value of the first QoS data field that indicates the class of service for the first RDMA packet, queueing the third RDMA packet in the first RDMA queue. ### Claim 2 (depends on 1) The method of claim 1, further comprising retrieving the first RDMA packet from the first RDMA queue, and retrieving the second RDMA packet from the second RDMA queue, according to a first weighting among the plurality of queues. ### Claim 3 (depends on 1) The method of claim 1, further comprising retrieving the first RDMA packet from the first RDMA queue, and retrieving the second RDMA packet from the second RDMA queue, according to a weighted-round-robin scheme. ### Claim 4 (depends on 1) The method of claim 1, wherein the plurality of queues includes a control queue, and wherein the method further comprises, by the networking device, retrieving a plurality of control packets from the control queue, wherein the retrieving the plurality of control packets has a strict priority over the retrieving the first RDMA packet and the retrieving the second RDMA packet. ### Claim 5 (depends on 4) The method of claim 4, wherein the plurality of queues includes a plurality of RDMA queues that includes the first RDMA queue and the second RDMA queue, and wherein the control queue is configured to have a lower bandwidth than any of the plurality of RDMA queues. ### Claim 6 (depends on 4) The method of claim 4, wherein the plurality of control packets includes at least one of a network control protocol packet or a congestion notification packet. ### Claim 7 (depends on 1) The method of claim 1, wherein the first QoS data field is a Differentiated Services Code Point (DSCP) data field of an Internet Protocol (IP) header of the first RDMA packet. ### Claim 8 (depends on 1) The method of claim 1, wherein each of the first and second RDMA packets is a RoCEv2 packet. ### Claim 9 (depends on 1) The method of claim 1, wherein each of the first and second RDMA packets is a Layer-3 encapsulated packet that is formatted in accordance with an overlay encapsulation protocol. ### Claim 10 (depends on 9) The method of claim 9, wherein the overlay encapsulation protocol is one among VxLAN, NVGRE, GENEVE, STT, and MPLS. ### Claim 11 (depends on 9) The method of claim 9, wherein the method further comprises: decapsulating the first RDMA packet to obtain a first decapsulated RDMA packet, and copying congestion indication information from at least one outer header of the first RDMA packet to the first decapsulated RDMA packet. ### Claim 12 (depends on 1) The method of claim 1, wherein the method further comprises: receiving a plurality of RDMA packets, each of the plurality of RDMA packets having a corresponding QoS data field; for each of the plurality of RDMA packets, queuing the RDMA packet in an RDMA queue of the plurality of queues, based on a value of the QoS data field of the RDMA packet that indicates a priority for the RDMA packet; and retrieving the plurality of RDMA packets from the plurality of queues, wherein the plurality of RDMA packets includes a plurality of packet flows, and wherein the method further comprises routing the plurality of packet flows according to a per-flow equal-cost multipath scheme. ### Claim 13 (depends on 1) The method according to claim 1, wherein the third RDMA packet is queued in the first RDMA queue based on the class of service of the third RDMA packet being the same as the class of service of the first RDMA packet. ### Claim 14 A networking device comprising: memory, the memory storing a plurality of queues; and processing circuitry coupled with the memory, the processing circuitry to: receive a first Remote Direct Memory Access (RDMA) packet having a first quality-of-service (QoS) data field and a first virtual local area network (VLAN) identifier, the first VLAN identifier indicating a first VLAN; based on a quality-of-service value of the first QoS data field that indicates a class of service for the first RDMA packet, queue the first RDMA packet in a first RDMA queue of the plurality of queues; receive a second RDMA packet having a second QoS data field and a second VLAN identifier, the second VLAN identifier indicating a second VLAN; based on a quality-of-service value of the second QoS data field that indicates a class of service for the second RDMA packet, queue the second RDMA packet in a second RDMA queue of the plurality of the queues, the second RDMA queue being different than the first RDMA queue; receive a third RDMA packet having a third QoS data field and a third VLAN identifier, the third VLAN identifier indicating a third VLAN that is different than the first VLAN; and based on a quality-of-service value of the third QoS data field that indicates a class of service for the third RDMA packet being the same as the quality-of-service value of the first QoS data field that indicates the class of service for the first RDMA packet, queue the third RDMA packet in the first RDMA queue. ### Claim 15 (depends on 14) The networking device of claim 14, wherein the plurality of queues includes a control queue, and wherein the processing circuitry is to retrieve a plurality of control packets from the control queue according to a strict priority over the retrieving the first RDMA packet and the retrieving the second RDMA packet. ### Claim 16 (depends on 15) The networking device of claim 15, wherein each of the first and second RDMA packets is a Layer-3 encapsulated packet that is formatted in accordance with an overlay encapsulation protocol. ### Claim 17 (depends on 16) The networking device of claim 16, wherein the processing circuitry is further to: decapsulate the first RDMA packet to obtain a first decapsulated RDMA packet, and copy congestion indication information from at least one outer header of the first RDMA packet to the first decapsulated RDMA packet. ### Claim 18 (depends on 14) The networking device of claim 14, wherein the networking device is a switch of a Clos network. ### Claim 19 A non-transitory computer-readable medium storing a plurality of instructions executable by one or more processors, the plurality of instructions comprising instructions that when executed by the one or more processors cause the one or more processors to: receive a first RDMA packet having a first quality-of-service (QoS) data field and a first virtual local area network (VLAN) identifier, the first VLAN identifier indicating a first VLAN; based on a quality-of-service value of the first QoS data field that indicates a class of service for the first RDMA packet, queue the first RDMA packet in a first RDMA queue of a plurality of queues; receive a second RDMA packet having a second QoS data field and a second VLAN identifier, the second VLAN identifier indicating a second VLAN; based on a quality-of-service value of the second QoS data field that indicates a class of service for the second RDMA packet, queue the second RDMA packet in a second RDMA queue of the plurality of the queues, the second RDMA queue being different than the first RDMA queue; receive a third RDMA packet having a third QoS data field and a third VLAN identifier, the third VLAN identifier indicating a third VLAN that is different than the first VLAN; and based on a quality-of-service value of the third QoS data field that indicates a class of service for the third RDMA packet being the same as the quality-of-service value of the first QoS data field that indicates the class of service for the first RDMA packet, queue the third RDMA packet in the first RDMA queue. ### Claim 20 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein the plurality of queues includes a control queue, and wherein the plurality of instructions further comprises instructions that when executed by the one or more processors cause the one or more processors to retrieve a plurality of control packets from the control queue according to a strict priority over the retrieving the first RDMA packet and the retrieving the second RDMA packet. ### Claim 21 (depends on 19) The non-transitory computer-readable medium of claim 19, wherein the first RDMA packet is for a first tenant among a plurality of tenants and the second RDMA packet is for a second tenant among the plurality of tenants, and wherein the first VLAN is a VLAN of a first compute instance among a plurality of compute instances executing on a host machine, and the second VLAN is a VLAN of a second compute instance among the plurality of compute instances.