# US 2006/0221974 A1 · Method and apparatus for dynamic load balancing over a network link bundle

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2006/0221974 A1
- Document ID: US20060221974A1
- Status: Published application
- Kind: A1
- Application number: 11/097,798
- Filed: 2005-04-02
- Published: 2006-10-05
- Earliest claimed: 2005-04-02
- Assignees: Cisco Technology, Inc.
- Inventors: Stephen Hilla; Kenneth Potter; John Marshall
- Web page: https://patentgrep.com/patent/US20060221974A1

## CPC

- H04L 47/10: Flow control; Congestion control
- H04L 45/60: Router architectures
- H04L 47/125: by balancing the load, e.g. traffic engineering
- H04L 47/2441: relying on flow classification, e.g. using integrated services [IntServ]
- H04L 47/266: {Stopping or restarting the source, e.g. X-on or X-off}
- H04L 47/30: in combination with information about buffer occupancy at either end or at transit nodes
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Techniques for distributing data packets over a network link bundle include storing an output data packet in a data flow queue based on a flow identification associated with the output data packet. The flow identification indicates a set of one or more data packets, including the output data packet, which are to be sent in the same sequence as received. State data is also received. The state data indicates a physical status of a first port of multiple active egress ports that are connected to a corresponding bundle of communication links with one particular network device. A particular data flow queue is determined based at least in part on the state data. A next data packet is directed from the particular data flow queue to a second port of the active egress ports. These techniques allow a more efficient use of a network link bundle.

## Claims

### Claim 1

A method for distributing data packets over a bundle of communication links between two network devices, comprising the steps of:
storing an output data packet in a data flow queue of a plurality of data flow queues based on a flow identification associated with the output data packet, wherein the flow identification indicates a set of one or more data packets including the output data packet that are to be sent in the same sequence as received;
receiving state data that indicates a physical status of a first port of a plurality of active egress ports connected to a bundle of communication links with one particular network device;
determining a particular data flow queue based at least in part on the state data; and
directing a next data packet from the particular data flow queue to a second port of the plurality of active egress ports.

### Claim 2 (depends on 1)

A method as recited in claim 1, wherein the second port is the same as the first port.

### Claim 3 (depends on 1)

A method as recited in claim 1, said step of receiving state data further comprising receiving state data that indicates a physical status of a first plurality of ports of the plurality of active egress ports, wherein the first plurality of ports includes the first port.

### Claim 4 (depends on 1)

A method as recited in claim 1, wherein:
said step of receiving state data further comprises receiving state data that indicates a buffer for the first port is too full to accept another data packet; and
said step of determining the particular data flow queue further comprises determining a data flow queue that is not directed to the first port.

### Claim 5 (depends on 1)

A method as recited in claim 1, wherein:
said step of receiving state data further comprises receiving state data that indicates a pause message has been received at the first port from the particular network device;
the pause message includes data that indicates that the particular network device is not ready to accept another data packet; and
said step of determining the particular data flow queue further comprises determining a data flow queue that is not directed to the first port.

### Claim 6 (depends on 5)

A method as recited in claim 5, said step of receiving state data that indicates the pause message has been received further comprises receiving state data that indicates the pause message has been received within a time interval during which the pause message is effective.

### Claim 7

A method for distributing data packets over a bundle of communication links between two network devices, comprising the steps of:
receiving first commitment data that indicates an amount of data in a data flow queue directed to a buffer for a port of a plurality of active egress ports connected to a bundle of communication links with one particular network device, wherein
a plurality of buffers for the plurality of active egress ports are different from a plurality of data flow queues that include the data flow queue, and
a plurality of data packets in a data flow queue are to be received at the particular network device in a same sequence as stored in the data flow queue;
determining a particular port of the plurality of active egress ports for a data flow queue that is not already directed to any buffer of the plurality of buffers based at least in part on the first commitment data; and
directing a particular output data packet from a data flow queue that is not already directed to any buffer to a buffer for the particular port.

### Claim 8 (depends on 7)

A method as recited in claim 7, further comprising the steps of:
receiving particular size data that indicates an amount of data in the particular data flow queue; and
storing third commitment data that indicates an amount of data in the particular data flow queue directed to the buffer for the particular port based on the particular size data.

### Claim 9 (depends on 7)

A method as recited in claim 7, said step of determining the particular port further comprising the step of determining that the amount of data already directed to the buffer for the particular port is no greater than the amount of data already directed to the buffer of any other port:

### Claim 10

A method for distributing data packets over a bundle of communication links between two network devices, comprising the steps of:
receiving a current data packet based on a data packet received at an ingress port;
receiving state data that indicates physical state of a first port of a plurality of active egress ports connected to a bundle of communication links with one particular network device other than a fill level for a buffer for the first port;
determining whether the state data indicates the first port is unsuitable for receiving additional data; and
if it is determined that the state data indicates the first port is unsuitable for receiving additional data, then directing the current data packet to a second port of the plurality of active egress ports, which second port is different from the first port.

### Claim 11 (depends on 10)

A method as recited in claim 10, wherein said state data indicates whether a pause message has been received at the first port, wherein the pause message includes data that indicates that the particular network device is not ready to accept another data packet.

### Claim 12

An apparatus for distributing data packets over a bundle of communication links between two network devices, comprising:
means for storing an output data packet in a data flow queue of a plurality of data flow queues based on a flow identification associated with the output data packet, wherein the flow identification indicates a set of one or more data packets including the output data packet that are to be sent in the same sequence as received;
means for receiving state data that indicates a physical status of a first port of a plurality of active egress ports connected to a bundle of communication links with one particular network device;
means for determining a particular data flow queue based at least in part on the state data; and
means for directing a next data packet from the particular data flow queue to a second port of the plurality of active egress ports.

### Claim 13

An apparatus for distributing data packets over a bundle of communication links between two network devices, comprising:
means for receiving first commitment data that indicates an amount of data in a data flow queue directed to a buffer for a port of a plurality of active egress ports connected to a bundle of communication links with one particular network device, wherein
a plurality of buffers for the plurality of active egress ports are different from a plurality of data flow queues that include the data flow queue, and
a plurality of data packets in a data flow queue are to be received at the particular network device in a same sequence as stored in the data flow queue;
means for receiving an output data packet from a particular data flow queue that is not directed to any buffer of the plurality of buffers;
mean for determining a particular port of the plurality of active egress ports based at least in part on the first commitment data; and
means for directing the particular output data packet to a buffer for the particular port.

### Claim 14

An apparatus for distributing data packets over a bundle of communication links between two network devices, comprising:
means for receiving a current data packet based on a data packet received at an ingress port;
means for receiving state data that indicates physical state of a first port of a plurality of active egress ports connected to a bundle of communication links with one particular network device other than a fill level for a buffer for the first port;
means for determining whether the state data indicates the first port is unsuitable for receiving additional data; and
means for directing the current data packet to a second port of the plurality of active egress ports, which second port is different from the first port, if it is determined that the state data indicates the first port is unsuitable for receiving additional data.

### Claim 15

An apparatus for distributing data packets over a bundle of communication links between two network devices, comprising:
an ingress network interface that is coupled to a network for receiving from the network an inbound data packet;
an egress network interface that includes a plurality of active egress ports coupled to a bundle of communication links with one particular network device for sending a current data packet to that particular device;
a computer-readable medium for storing information;
one or more processors; and
one or more sequences of instructions stored on the computer-readable medium, which, when executed by the one or more processors, causes the one or more processors to carry out the steps of:
storing an output data packet in a data flow queue of a plurality of data flow queues based on a flow identification associated with the output data packet, wherein
the output data packet is based on the inbound data packet, and
the flow identification indicates a set of one or more data packets including the output data packet that are to be sent in the same sequence as received on the ingress network interface;
receiving state data that indicates a physical status of a first port of the plurality of active egress ports;
determining a particular data flow queue based at least in part on the state data; and
directing a next data packet from the particular data flow queue to a second port of the plurality of active egress ports.

### Claim 16 (depends on 15)

An apparatus as recited in claim 15, wherein the second port is the same as the first port.

### Claim 17 (depends on 15)

An apparatus as recited in claim 15, said step of receiving state data further comprising receiving state data that indicates a physical status of a first plurality of ports of the plurality of active egress ports, wherein the first plurality of ports includes the first port.

### Claim 18 (depends on 15)

An apparatus as recited in claim 15, wherein:
said step of receiving state data further comprises receiving state data that indicates a buffer for the first port is too full to accept another data packet; and
said step of determining the particular data flow queue further comprises determining a data flow queue that is not directed to the first port.

### Claim 19 (depends on 15)

An apparatus as recited in claim 15, wherein:
said step of receiving state data further comprises receiving state data that indicates a pause message has been received at the first port from the particular network device;
the pause message includes data that indicates that the particular network device is not ready to accept another data packet; and
said step of determining the particular data flow queue further comprises determining a data flow queue that is not directed to the first port.

### Claim 20 (depends on 19)

An apparatus as recited in claim 19, said step of receiving state data that indicates the pause message has been received further comprises receiving state data that indicates the pause message has been received within a time interval during which the pause message is effective.

### Claim 21

An apparatus for distributing data packets over a bundle of communication links between two network devices, comprising:
an ingress network interface that is coupled to a network for receiving from the network an inbound data packet;
an egress network interface that includes a plurality of active egress ports coupled to a bundle of communication links with one particular network device for sending a current data packet to that particular device;
a computer-readable medium for storing information;
one or more processors; and
one or more sequences of instructions stored on the computer-readable medium, which, when executed by the one or more processors, causes the one or more processors to carry out the steps of:
receiving first commitment data that indicates an amount of data in a data flow queue directed to a buffer for a port of the plurality of active egress ports, wherein
a plurality of buffers for the plurality of active egress ports are different from a plurality of data flow queues that include the data flow queue, and
a plurality of data packets in a data flow queue are to be received at the particular network device in a same sequence as stored in the data flow queue;
receiving an output data packet from a particular data flow queue that is not directed to any buffer of the plurality of buffers;
determining a particular port of the plurality of active egress ports for a data flow queue that is not already directed to any buffer of the plurality of buffers based at least in part on the first commitment data; and
directing a particular output data packet from a data flow queue that is not already directed to any buffer to a buffer for the particular port.

### Claim 22 (depends on 21)

An apparatus as recited in claim 21, wherein execution of the one or more sequences of instructions further causes the one or more processors to carry out the steps of:
receiving particular size data that indicates an amount of data in the particular data flow queue; and
storing third commitment data that indicates an amount of data in the particular data flow queue directed to the buffer for the particular port based on the particular size data.

### Claim 23 (depends on 21)

An apparatus as recited in claim 21, said step of determining the particular port further comprising the step of determining that the amount of data already directed to the buffer for the particular port is no greater than the amount of data already directed to the buffer of any other port:

### Claim 24

An apparatus for distributing data packets over a bundle of communication links between two network devices, comprising:
an ingress network interface that is coupled to a network for receiving from the network an inbound data packet;
an egress network interface that includes a plurality of active egress ports coupled to a bundle of communication links with one particular network device for sending a current data packet to that particular device;
a computer-readable medium for storing information;
one or more processors; and
one or more sequences of instructions stored on the computer-readable medium, which, when executed by the one or more processors, causes the one or more processors to carry out the steps of:
receiving a current data packet based on the inbound data packet;
receiving state data that indicates physical state of a first port of the plurality of active egress ports other than a fill level for a buffer for the first port;
determining whether the state data indicates the first port is unsuitable for receiving additional data; and
if it is determined that the state data indicates the first port is unsuitable for receiving additional data, then directing the current data packet to a second port of the plurality of active egress ports, which second port is different from the first port.

### Claim 25 (depends on 24)

An apparatus as recited in claim 24, wherein said state data indicates whether a pause message has been received at the first port, wherein the pause message includes data that indicates that the particular network device is not ready to accept another data packet.
