# US 11,444,881 B2 · System and method for supporting use of forward and backward congestion notifications in a private fabric in a high performance computing environment

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,444,881 B2
- Document ID: US11444881B2
- Status: Granted
- Kind: B2
- Application number: 16/872,043
- Filed: 2020-05-11
- Published: 2021-05-20
- Granted: 2022-09-13
- Earliest claimed: 2019-11-19
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Bjørn Dag Johnsen; Ajoy Siddabathuni; Avneesh Pant; Amarnath Jolad
- Web page: https://patentgrep.com/patent/US11444881B2

## CPC

- H04L 49/358: {Infiniband Switches}
- H04L 41/12: Discovery or management of network topologies
- H04L 41/5003: Managing SLA; Interaction between SLA and QoS
- H04L 47/12: Avoiding congestion; Recovering from congestion
- H04L 47/17: Interaction among intermediate nodes, e.g. hop by hop
- H04L 47/24: Traffic characterised by specific attributes, e.g. priority or QoS
- H04L 47/2433: {Allocation of priorities to traffic types}
- H04L 47/265: sent by intermediate network nodes
- H04L 47/30: in combination with information about buffer occupancy at either end or at transit nodes
- H04L 47/33: using forward notification
- H04L 47/805: {QOS or priority aware}
- H04W 28/0983: {for optimizing bandwidth or throughput}
- G06F 9/45558: {Hypervisor-specific management and integration aspects}
- G06F 15/17331: {Distributed shared memory [DSM], e.g. remote direct memory access [RDMA]}
- G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing}
- G06F 2009/45579: {I/O management, e.g. providing access to device drivers or storage}
- G06F 2009/45583: {Memory management, e.g. access or allocation}
- G06F 2009/45595: {Network integration; Enabling network access in virtual machine instances}
- H04L 49/70: {Virtual switches}
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods for using multiple CE (congestion experienced) flags in both FECN (forward explicit congestion notification) and BECN (backward explicit congestion notification) in a high performance computing environment. An exemplary method can provide a first subnet comprising a plurality of switches, a plurality of host channel adapters, and a plurality of end nodes. The method can receive, at an end node attached to a host channel adapter, an ingress packet from a remote end node, wherein the ingress packet traversed at least a portion of the first subnet prior to being received at the end node. The method can, on receiving the ingress packet, send a response message from the end node attached to the host channel adapter to the remote end node, the response message indicating that the ingress packet experienced congestion during the traversal of the at least a portion of the first subnet.

## Claims

### Claim 1

A system for using multiple CE (congestion experienced) flags in both FECN (forward explicit congestion notification) and BECN (backward explicit congestion notification) in a high performance computing environment, comprising:
a computer comprising one or more microprocessors;
a first subnet provided at the computer, the first subnet comprising
a plurality of switches, the plurality of switches comprising at least a leaf switch, wherein each of the plurality of switches comprise a plurality of switch ports,
a plurality of host channel adapters, wherein each of the host channel adapters comprise at least one host channel adapter port, and wherein the plurality of host channel adapters is interconnected via the plurality of switches, and
a plurality of end nodes, including a plurality of virtual machines;
wherein an ingress packet is received at an end node attached to a host channel adapter from a remote end node, wherein the ingress packet traversed at least a portion of the first subnet along a path comprising a plurality of path stages prior to being received at the end node, wherein the ingress packet comprises a bit field, the bit field comprising a plurality of bits, indicating that the ingress packet experienced congestion at a first path stage of the plurality of path stages during the traversal of the at least a portion of the first subnet, wherein each bit of the bit field corresponds to a separate path stage the plurality of path stages, and wherein the indication of congestion at the first path stage is set within the bit field by a switch port associated with the first path stage;
wherein on receiving the ingress packet, the end node determines, by correlating an information of the path with the ingress packet, distinct switch ports at which the ingress packet experienced congestion, and sends a response message to the remote end node, the response message indicating that the ingress packet experienced congestion during the traversal of the at least a portion of the first subnet.

### Claim 2 (depends on 1)

The system of claim 1, wherein the response message comprises a bit field of the response message comprising a plurality of bits.

### Claim 3 (depends on 2)

The system of claim 2, wherein each path stage comprises one of an inter-switch link or a switch port.

### Claim 4 (depends on 3)

The system of claim 3, wherein each of the plurality of bits of the bit field of the response message correspond to a different path stage of the plurality of path stages, respectively.

### Claim 5 (depends on 4)

The system of claim 4, wherein a first bit of the bit field of the response message corresponding to the first path stage of the plurality of path stages is indicative of, within the response message, the ingress packet experiencing congestion at the first path stage.

### Claim 6 (depends on 5)

The system of claim 5, wherein the first bit of the bit field of the response message corresponding to the first path stage of the plurality of path stages comprises a CE flag.

### Claim 7 (depends on 6)

The system of claim 6, wherein the first subnet comprises an InfiniBand subnet.

### Claim 8

A method for using multiple CE (congestion experienced) flags in both FECN (forward explicit congestion notification) and BECN (backward explicit congestion notification) in a high performance computing environment, comprising:
providing, at one or more microprocessors, a first subnet, the first subnet comprising
a plurality of switches, the plurality of switches comprising at least a leaf switch, wherein each of the plurality of switches comprise a plurality of switch ports,
a plurality of host channel adapters, wherein each of the host channel adapters comprise at least one host channel adapter port, and wherein the plurality of host channel adapters is interconnected via the plurality of switches, and
a plurality of end nodes, including a plurality of virtual machines;
receiving, at an end node attached to a host channel adapter, an ingress packet from a remote end node, wherein the ingress packet traversed at least a portion of the first subnet along a path comprising a plurality of path stages prior to being received at the end node, wherein the ingress packet comprises a bit field, the bit field comprising a plurality of bits, indicating that the ingress packet experienced congestion at a first path stage of the plurality of path stages during the traversal of the at least a portion of the first subnet, wherein each bit of the bit field corresponds to a separate path stage the plurality of path stages, and wherein the indication of congestion at the first path stage is set within the bit field by a switch port associated with the first path stage;
on receiving the ingress packet, determining, by the end node, by correlating an information of the path with the ingress packet, district switch ports at which the ingress packet experienced congestion, and sending a response message from the end node attached to the host channel adapter to the remote end node, the response message indicating that the ingress packet experienced congestion during the traversal of the at least a portion of the first subnet.

### Claim 9 (depends on 8)

The method of claim 8, wherein the response message comprises a bit field of the response message comprising a plurality of bits.

### Claim 10 (depends on 9)

The method of claim 9, each path stage comprises one of an inter-switch link or a switch port.

### Claim 11 (depends on 10)

The method of claim 10, wherein each of the plurality of bits of the bit field of the response message correspond to a different path stage of the plurality of path stages, respectively.

### Claim 12 (depends on 11)

The method of claim 11, wherein a first bit of the bit field of the response message corresponding to the first path stage of the plurality of path stages is indicative of, within the response message, the ingress packet experiencing congestion at the first path stage.

### Claim 13 (depends on 12)

The method of claim 12, wherein the first bit of the bit field of the response message corresponding to the first path stage of the plurality of path stages comprises a CE flag.

### Claim 14 (depends on 13)

The method of claim 13, wherein the first subnet comprises an InfiniBand subnet.

### Claim 15

A non-transitory computer readable storage medium having instructions thereon for using multiple CE (congestion experienced) flags in both FECN (forward explicit congestion notification) and BECN (backward explicit congestion notification) in a high performance computing environment, which when read and executed cause a computer to perform steps comprising:
providing, at one or more microprocessors, a first subnet, the first subnet comprising
a plurality of switches, the plurality of switches comprising at least a leaf switch, wherein each of the plurality of switches comprise a plurality of switch ports,
a plurality of host channel adapters, wherein each of the host channel adapters comprise at least one host channel adapter port, and wherein the plurality of host channel adapters is interconnected via the plurality of switches, and
a plurality of end nodes, including a plurality of virtual machines;
receiving, at an end node attached to a host channel adapter, an ingress packet from a remote end node, wherein the ingress packet traversed at least a portion of the first subnet along a path comprising a plurality of path stages prior to being received at the end node, wherein the ingress packet comprises a bit field, the bit field comprising a plurality of bits, indicating that the ingress packet experienced congestion at a first path stage of the plurality of path stages during the traversal of the at least a portion of the first subnet, wherein each bit of the bit filed corresponds to a separate path stage the plurality of path stages, and wherein the indication of congestion at the first path stage is set within the bit field by a switch port associated with the first path stage;
on receiving the ingress packet, determining, by the end node, by correlating an information of the path with the ingress packet, district switch ports at which the ingress packet experienced congestion, and sending a response message from the end node attached to the host channel adapter to the remote end node, the response message indicating that the ingress packet experienced congestion during the traversal of the at least a portion of the first subnet.

### Claim 16 (depends on 15)

The non-transitory computer readable storage medium of claim 15, wherein the response message comprises a bit field of the response message comprising a plurality of bits.

### Claim 17 (depends on 16)

The non-transitory computer readable storage medium of claim 16, wherein each path stage comprises one of an inter-switch link or a switch port.

### Claim 18 (depends on 17)

The non-transitory computer readable storage medium of claim 17, wherein each of the plurality of bits of the bit field of the response message correspond to a different path stage of the plurality of path stages, respectively.

### Claim 19 (depends on 18)

The non-transitory computer readable storage medium of claim 18, wherein a first bit of the bit field of the response message corresponding to the first path stage of the plurality of path stages is indicative of, within the response message, the ingress packet experiencing congestion at the first path stage.

### Claim 20 (depends on 19)

The non-transitory computer readable storage medium of claim 19, wherein the first bit of the bit field of the response message corresponding to the first path stage of the plurality of path stages comprises a CE flag.