Receiver-based traffic scheduling for incast congestion management in high-performance AI/ML networks
AssigneeMeta Platforms, Inc.
InventorsZeeshan Altaf Lokhandwala, Arvind Srinivasan
Life of this patent
Estimating the term…
Abstract
A system for Remote Direct Memory Access incast congestion management is provided. The system may access one or more receive work queue entities (RWQEs) associated with one or more send work queue entities (SWQEs), determine a transmission rate based on an available bandwidth and a number of transmitter devices associated with the one or more SWQEs, and transmit a rate control notification (RCN) message to a transmitter device associated with a send work queue entity of the one or more SWQEs, wherein the rate control notification message comprises the determined transmission rate.
Claims17
A method comprising:
- accessing one or more receive work queue entities (RWQEs) associated with one or more send work queue entities (SWQEs);
- determining a transmission rate based on an available bandwidth and a number of transmitter devices associated with the one or more SWQEs;
- transmitting a rate control notification (RCN) message to a transmitter device associated with a send work queue entity of the one or more SWQEs, wherein the rate control notification message comprises the determined transmission rate;
- receiving, from the one or more SWQEs associated with the transmitter device, a message at the determined transmission rate;
- deriving a second transmission rate based on the message;
- and
- re-transmitting an RCN message to the transmitter device in response to the second transmission rate being different than the determined transmission rate.
An apparatus comprising:
- one or more processors;
- and
- at least one memory storing instructions, that when executed by the one or more processors, cause the apparatus to:
- access one or more receive work queue entities (RWQEs) associated with one or more send work queue entities (SWQEs);
- determine a transmission rate based on an available bandwidth and a number of transmitter devices associated with the one or more SWQEs;
- transmit a rate control notification (RCN) message to a transmitter device associated with a send work queue entity of the one or more SWQEs, wherein the rate control notification message comprises the determined transmission rate;
- remove from the one or more RWQEs a receive work queue entity, in response to the receive work queue entity indicating completion;
- remove from the one or more SWQEs a send work queue entity associated with the receive work queue entity, in response to the receive work queue entity indicating completion;
- determine a new transmission rate based on the available bandwidth and the number of transmitter devices;
- and
- transmit one or more second RCN messages to the transmitter device, wherein the one or more second RCN messages comprise the new transmission rate.
A computer-readable medium storing instructions that, when executed, cause:
- accessing one or more receive work queue entities (RWQEs) associated with one or more send work queue entities (SWQEs);
- determining a transmission rate based on an available bandwidth and a number of transmitter devices associated with the one or more SWQEs;
- transmitting a rate control notification (RCN) message to a transmitter device associated with a send work queue entity of the one or more SWQEs, wherein the rate control notification message comprises the determined transmission rate;
- receiving, from the one or more SWQEs associated with the transmitter device, a message at the determined transmission rate;
- deriving a second transmission rate based on the message;
- and
- re-transmitting an RCN message to the transmitter device in response to the second transmission rate being different than the determined transmission rate.
Dependencies are inferred from the claim text.