# US 10,873,529 B2 · Method and apparatus for low latency data center network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,873,529 B2 - Document ID: US10873529B2 - Status: Granted - Kind: B2 - Application number: 15/829,319 - Filed: 2017-12-01 - Published: 2019-06-06 - Granted: 2020-12-22 - Earliest claimed: 2017-12-01 - Assignees: Futurewei Technologies, Inc. - Applicants: Futurewei Technologies, Inc. - Inventors: Fangping Liu; Zhenjiang Li; Serhat Nazim Avci - Web page: https://patentgrep.com/patent/US10873529B2 ## CPC - H04L 47/25: with rate being modified by the source upon detecting a change of network conditions - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Methods and apparatus for determining and adjusting traffic rate limits to reduce queuing delays and latency in a data center network. In various embodiments, a network controller receives network topology information and traffic data relating to flows of the network. The traffic data includes traffic matrix information, flow path information, and pause frame distribution information. Based on this information, the network controller identifies server network node interfaces that are the source of a pause frame. Flows associated with such network node interfaces are also identified and ranked based on traffic volume. For ranked flows meeting predetermined criteria, reduced traffic rate limits are calculated and sent to associated server network node interfaces. In some embodiments, the reduced traffic rate limits are incrementally recovered when normal traffic resumes on a paused link. In further embodiments, traffic rate limits are also adjusted for flows of highly utilized and underutilized network node interfaces. ## Claims ### Claim 1 A computer-implemented method for reducing latency in a network having a plurality of servers, the method comprising: receiving traffic data relating to flows of the network, the traffic data including traffic matrix information, flow path information, and pause frame distribution information; identifying, based on network topology information and the pause frame distribution information, one or more network node interfaces of the plurality of servers that are a source of a pause frame; identifying flows in the traffic matrix information which pass through the one or more network node interfaces; determining, based on the traffic data and predetermined criteria, a reduced traffic rate limit for each of a plurality of the identified flows, the determining further comprising ranking the identified flows based on respective flow traffic volume, and decreasing a current traffic rate limit for each of a ranked flows in which the flow traffic volume exceeds the predetermined criteria; identifying, based on the network topology information and the traffic data, one or more additional network node interfaces having a utilization rate greater than a predetermined threshold; reducing an associated traffic rate limit for flows passing through the one or more additional network node interfaces with the utilization rate greater than the predetermined threshold; determining adjusted traffic rate limits by incrementally increasing the reduced traffic rate limit for flows of a network node interface, of the one or more network node interfaces, identified as no longer a source of pause frame based on the reduced traffic rate limit; determining further adjusted traffic rate limits by incrementally decreasing the reduced traffic rate limit for the identified flows of the network node interface when the network node interface is the source of a pause frame based on the adjusted traffic rate limits; and determining further adjusted traffic rate limits by incrementally increasing the reduced traffic rate limit for the identified flows of the network node interface when the network node interface is not the source of a pause frame based on the adjusted traffic rate limits. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the predetermined criteria comprises: ranking the flows of an identified network node interface based on respective flow traffic volume values of the flows; and identifying one or more of the ranked flows having a cumulative traffic volume that constitutes a predetermined portion of a total flow traffic volume of the identified network node interface. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein decreasing the current traffic rate limit by a percentage basis. ### Claim 4 (depends on 1) The computer-implemented method of claim 1 further comprising: receiving updated pause frame distribution information; identifying, based on the network topology information and the updated pause frame distribution information, the network node interface, of the one or more network node interfaces, that is no longer the source of the pause frame based on the reduced traffic rate limit; and sending the adjusted traffic rate limits for application to the network node interface. ### Claim 5 (depends on 4) The computer-implemented method of claim 4 further comprising: receiving further updated pause frame distribution information; determining whether the further updated pause frame distribution information indicates that the network node interface is the source of the pause frame based on the adjusted traffic rate limits; and sending the further adjusted traffic rate limits for application to the network node interface. ### Claim 6 (depends on 1) The computer-implemented method of claim 1 further comprising: identifying, based on the traffic data, additional flows of the additional network node interfaces; determining additional reduced traffic rate limit for one or more of the additional flows of the additional node interfaces; and sending the additional reduced traffic rate limit for application to the additional network node interfaces. ### Claim 7 (depends on 1) The computer-implemented method of claim 1 further comprising: identifying, based on the network topology information and the traffic data, the additional network node interfaces of the plurality of servers having the utilisation rate less than the predetermined threshold; identifying, based on the traffic data, additional flows of the additional network node interfaces, wherein each network node interface of an additional flow is an additional network node interface; determining increased traffic rate limits for the additional flows of the additional network node interfaces; and sending the increased traffic rate limits for application to the additional network node interfaces. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, further comprising: identifying, based on the traffic data, initial flows of the plurality of servers; determining, based on the traffic data, an initial traffic rate limit for each of the initial flows; and sending the initial rate limits for application to the plurality of servers. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the network is a data center network. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the method is repeated on a periodic basis. ### Claim 11 A device comprising: a memory storage comprising instructions; at least one network interface; and one or more processors in communication with the memory storage, wherein the one or more processors execute the instructions to: receive, via the at least one network interface, traffic data relating to flows of the network, the traffic data including traffic matrix information, flow path information, and pause frame distribution information; identify, based on network topology information and the pause frame distribution information, one or more network node interfaces of a plurality of servers that are a source of a pause frame; identify flows in the traffic matrix information which pass through the one or more network node interfaces; determine, based on the traffic data and predetermined criteria, a reduced traffic rate limit for each of a plurality of the identified flows, and further comprising ranking the identified flows based on respective flow traffic volume, and decreasing a current traffic rate limit for each of the ranked flows in which the flow traffic volume exceeds the predetermined criteria; identify, based on the network topology information and the traffic data, one or more additional network node interfaces having a utilization rate greater than a predetermined threshold; reduce an associated traffic rate limit for flows passing through the one or more additional network node interfaces with the utilization rate greater than the predetermined threshold; determine adjusted traffic rate limits by incrementally increasing the reduced traffic rate limit for flows of a network node interface, of the one or more network node interfaces, identified as no longer a source of pause frame based on the reduced traffic rate limit; determine further adjusted traffic rate limits by incrementally decreasing the reduced traffic rate limit for the identified flows of the network node interface when the network node interface is a source of the pause frame based on the adjusted traffic rate limits; and determine further adjusted traffic rate limits by incrementally increasing the reduced traffic rate limit for the identified flows of the network node interface when the network node interface is not a source of the pause frame based on the adjusted traffic rate limits. ### Claim 12 (depends on 11) The device of claim 11, wherein the predetermined criteria comprises: ranking the flows of an identified network node interface based on respective flow traffic volume; and identifying one or more of the ranked flows having a cumulative traffic volume that constitutes a predetermined portion of a total flow traffic volume of the identified network node interface. ### Claim 13 (depends on 11) The device of claim 11, wherein the one or more processors further execute the instructions to: receive, via the at least one network interface, updated pause frame distribution information; identify, based on the network topology information and the updated pause frame distribution information, the network node interface, of the one or more network node interfaces, that is no longer the source of the pause frame based on the reduced traffic rate limit; and send, via the at least one network interface, the adjusted traffic rate limits for application to the network node interface. ### Claim 14 (depends on 13) The device of claim 13, wherein the one or more processors further execute the instructions to: receive, via the at least one network interface, further updated pause frame distribution information; determine whether the further updated pause frame distribution information indicates that the network node interface is a source of a pause frame based on the adjusted traffic rate limits; and send, via the at least one network interface, the further adjusted traffic rate limits for application to the network node interface for application to the additional network node interfaces. ### Claim 15 (depends on 11) The device of claim 11, wherein the one or more processors further execute the instructions to: identify, based on the network topology information and the traffic data, additional network node interfaces of the plurality of servers having a utilisation rate greater than a first predetermine d threshold; identify, based on the traffic data, additional flows of the additional network node interfaces; determine additional reduced traffic rate limit for one or more of the additional flows of the additional network node interfaces; and send, via the at least one network interface, the additional reduced traffic rate limit for application to the additional network node interfaces. ### Claim 16 (depends on 11) The device of claim 11, wherein the one or more processors further execute the instructions to: identify, based on the traffic data, additional flows of the additional network node interfaces, wherein each network node interface of an additional flow is an additional network node interface; determine increased traffic rate limits for the additional flows of the additional network node interfaces; and send, via the at least one network interface, the increased traffic rate limits for application to the additional network node interfaces. ### Claim 17 (depends on 11) The device of claim 11, wherein the one or more processors execute the instructions on a periodic basis. ### Claim 18 (depends on 11) The device of claim 11, wherein the one or more processors further execute the instructions to: identify, based on the traffic data, initial flows of the plurality of servers; determine, based on the traffic data, an initial traffic rate limit for each of the initial flows; and send, via the at least one network interface, the initial rate limits for application by the plurality of servers.