# US 2013/0297798 A1 · Two level packet distribution with stateless first level packet distribution to a group of servers and stateful second level packet distribution to a server within the group

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0297798 A1
- Document ID: US20130297798A1
- Status: Published application
- Kind: A1
- Application number: 13/464,608
- Filed: 2012-05-04
- Published: 2013-11-07
- Earliest claimed: 2012-05-04
- Assignees: Telefonaktiebolaget L M Ericsson (publ)
- Inventors: MUSTAFA ARISOYLU; Abhishek Arora; Prashant Anand
- Web page: https://patentgrep.com/patent/US20130297798A1

## CPC

- H04L 67/1014: based on the content of a request
- H04L 67/1001: for accessing one among a plurality of replicated servers
- H04L 67/1006: with static server selection, e.g. the same server being selected for a specific client
- H04L 67/1023: based on a hash applied to IP addresses or costs
- H04L 67/1027: Persistence of sessions during load balancing
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

A method, in one or more network elements that are in communication between clients that transmit packets and servers, of distributing the packets among the servers which are to process the packets. Stickiness of flows to servers assigned to process them is provided. A packet of a flow is received at a static first level packet distribution module. A group of servers is statically selected for the packet of the flow with the first level module. State that assigns the packet of the flow to the selected group of servers is not used. The packet of the flow is distributed to a distributed stateful second level packet distribution system. A server of the selected group is statefully selected with the second level system by accessing state that assigns processing of packets of the flow to the selected server. The packet of the flow is distributed to the selected server.

## Claims

### Claim 1

A method, which is performed in one or more network elements that are in communication between clients that transmit packets and servers, of distributing the packets among the servers which are to process the packets, while providing stickiness of flows of the packets to servers assigned to process the packets of the flows, the method comprising the steps of:
receiving a packet of a flow at a static first level packet distribution module;
statically selecting a group of servers, from among a plurality of groups of the servers, for the packet of the flow with the static first level packet distribution module, wherein statically selecting the group of servers does not involve using state that assigns the packet of the flow to the selected group of servers;
distributing the packet of the flow to a distributed stateful second level packet distribution system;
statefully selecting a server of the selected group of servers that is to process the packet of the flow with the distributed stateful second level packet distribution system by accessing state that assigns processing of packets of the flow to the selected server; and
distributing the packet of the flow to the selected server.

### Claim 2 (depends on 1)

The method of claim 1, wherein the distributed stateful second level packet distribution system is implemented on hardware that is distinct from hardware used to implement the static first level packet distribution module.

### Claim 3 (depends on 1)

The method of claim 1, wherein the distributed stateful second level packet distribution system is implemented on hardware that includes hardware used to implement the selected group of servers.

### Claim 4 (depends on 1)

The method of claim 1, wherein the step of distributing the packet of the flow to the distributed stateful second level packet distribution system comprises a step of distributing the packet of the flow to each of a plurality of distributed stateful second level packet distribution modules that each correspond to a different server of the selected group of servers.

### Claim 5 (depends on 4)

The method of claim 4, wherein each of the distributed stateful second level packet distribution modules performs the steps of:
determining that the packet is not for a new flow by examining a header of the packet;
determining whether a flow identifier corresponding to the flow for the packet is included in a flow table of the distributed stateful second level packet distribution module that is making the determination, wherein the flow table assigns flow identifiers to the server that corresponds to the distributed stateful second level packet distribution module that is making the determination; and
one of: (1) causing the packet to be dropped if the flow identifier is not included in the flow table; and (2) causing the packet to be accepted for processing by the corresponding server if the flow identifier is included in the flow table.

### Claim 6 (depends on 5)

The method of claim 5, wherein each of the distributed stateful second level packet distribution modules performs the steps of:
receiving a second packet;
determining that the second packet is for a new flow by examining a header of the second packet;
selecting a server to process the second packet with a packet distribution algorithm, wherein the packet distribution algorithm is operable to select the server consistently with packet distribution algorithms of other distributed stateful second level packet distribution modules for the selected group;
determining whether the server selected to process the second packet corresponds to the distributed stateful second level packet distribution module that selected the server; and
one of (1) causing the packet to be dropped if the selected server does not correspond to the distributed stateful second level packet distribution module that selected the server; and (2) recording a new flow identifier in a flow table of the packet distribution module that selected the server and causing the packet to be accepted for processing by the selected server if the selected server does correspond to the distributed stateful second level packet distribution module that selected the server.

### Claim 7 (depends on 4)

The method of claim 4, wherein each of the distributed stateful second level packet distribution modules performs the steps of:
determining whether a flow identifier corresponding to the flow for the packet is included in a flow table of the distributed stateful second level packet distribution module, wherein the flow table assigns flow identifiers to the server that corresponds to the distributed stateful second level packet distribution module; and
one of: (1) causing the packet to be accepted for processing by the corresponding server if the flow identifier is included in the flow table; and (2) causing the packet to be dropped if the flow identifier is not included in the flow table and the flow identifier is included in an exists-on-others flow table of the distributed stateful second level packet distribution module, wherein the exists-on-others flow table lists flow identifiers assigned to servers of the group other than the server corresponding to the distributed stateful second level packet distribution module.

### Claim 8 (depends on 1)

The method of claim 1, wherein the distributed stateful second level packet distribution system includes a plurality of distributed stateful second level packet distribution modules that each correspond to a different server of the selected group of servers, and wherein a distributed stateful second level packet distribution module performs the steps of:
sending a message to other distributed stateful second level packet distribution modules that correspond to the servers of the selected group, wherein the message indicates a state and a load of a server corresponding to the distributed stateful second level packet distribution module;
receiving messages from the other distributed stateful second level packet distribution modules, wherein the received messages indicate states and loads of the servers corresponding to the other distributed stateful second level packet distribution modules; and
adjusting a packet distribution algorithm used by the distributed stateful second level packet distribution module based on the states and loads indicated in the received messages.

### Claim 9 (depends on 1)

The method of claim 1, wherein the distributed stateful second level packet distribution system includes at least one distributed stateful second level packet distribution module corresponding to the selected group of servers and at least one distributed stateful second level packet distribution module corresponding to a second group of servers, and wherein the method further comprises the steps of:
receiving a message at the at least one distributed stateful second level packet distribution module corresponding to the selected group of servers from the at least one distributed stateful second level packet distribution module corresponding to a second group of servers, wherein the received message indicates at least one load associated with the second group of servers; and
re-assigning a server of the second group to the selected group based at least in part on the at least one load indicted in the received message and at least one load associated with the selected group of servers.

### Claim 10 (depends on 1)

The method of claim 1, wherein the method is performed in a single network element having a plurality of line cards and a plurality of cards selected from control cards, service cards, and resource cards, wherein hardware used to implement the static first level packet distribution module comprises hardware of the line cards, and wherein hardware across which the distributed stateful second level packet distribution system is distributed comprises hardware of the cards selected from the control cards, the service cards, and the resource cards.

### Claim 11 (depends on 1)

The method of claim 1, further comprising the steps of:
receiving a second packet of a second flow at the static first level packet distribution module, wherein the second flow is a larger volume flow;
statefully selecting a server for the second packet of the second flow with the static first level packet distribution module, wherein statefully selecting the server for the second packet includes using state that assigns larger volume flows to the servers; and
distributing the second packet of the second flow to the selected server.

### Claim 12 (depends on 1)

The method of claim 1, wherein each of the plurality of groups of servers form a different virtual local area network (VLAN).

### Claim 13 (depends on 1)

The method of claim 1, wherein statefully selecting the server of the selected group of servers that is to process the packet of the flow comprises selecting the server to balance loads on the servers of the selected group, wherein statically selecting the group of servers comprises performing a hash function with information from the packet to statically select the group of servers, wherein each of the servers of the selected group comprises one of a different server blade, a different control card, a different service card, a different processor, and a different processor core.

### Claim 14 (depends on 1)

The method of claim 1, further adding a server to the selected group of servers, wherein adding the server to the selected group does not upset stickiness of existing flows to the servers of the selected group.

### Claim 15 (depends on 1)

The method of claim 1, further comprising, after processing the packet with the selected server:
statically selecting a second group of servers from among the plurality of groups of the servers; and
distributing the packet to the second group of servers.

### Claim 16

A multi-level packet distribution system, the multi-level packet distribution system configured to be deployed on one or more network elements in a network between clients that transmit packets and servers, of distributing the packets among the servers, while providing stickiness of flows of the packets to the servers, the multi-level packet distribution system comprising:
a static first level packet distribution module configured to receive a packet of a flow, the static first level packet distribution module operable to statically select a group of servers, from among a plurality of groups of the servers, for the packet of the flow, wherein statically selecting the group of servers does not involve using state that assigns the packet of the flow to the selected group of servers; and
a distributed stateful second level packet distribution system in communication with the static first level packet distribution module, wherein the distributed stateful second level packet distribution system includes a plurality of distributed stateful second level packet distribution modules, the distributed stateful second level packet distribution system operable to statefully select a server of the selected group of servers that is to process the packet of the flow, wherein statefully selecting the server involves using state that assigns the flow to the selected server, the distributed stateful second level packet distribution system operable to distribute the packet to the selected server.

### Claim 17 (depends on 16)

The multi-level packet distribution system of claim 16, wherein the distributed stateful second level packet distribution system is implemented on hardware that is distinct from hardware used to implement the static first level packet distribution module.

### Claim 18 (depends on 16)

The multi-level packet distribution system of claim 16, wherein the distributed stateful second level packet distribution system is implemented on hardware that includes hardware used to implement the selected group of servers.

### Claim 19 (depends on 16)

The multi-level packet distribution system of claim 16, wherein the multi-level packet distribution system is configured to be deployed in a single network element having a plurality of line cards and a plurality of cards selected from control cards, service cards, and resource cards, wherein hardware used to implement the static first level packet distribution module comprises hardware of the line cards, and wherein the distributed stateful second level packet distribution system is distributed across hardware of the cards selected from the control cards, the service cards, and the resource cards.

### Claim 20 (depends on 16)

The multi-level packet distribution system of claim 16, wherein the plurality of distributed stateful second level packet distribution modules comprise a plurality of distributed stateful second level packet distribution modules that each correspond to a different server of the selected group of servers.

### Claim 21 (depends on 20)

The multi-level packet distribution system of claim 20, wherein a distributed stateful second level packet distribution module corresponding to a server comprises:
a packet examination module operable to determine that the packet is not for a new flow by examining a header of the packet;
a flow table that assigns flow identifiers to the server corresponding to the distributed stateful second level packet distribution module;
a flow table checker module operable to determine whether a flow identifier corresponding to the flow for the packet is included in the flow table; and
a determination module operable to one of: (1) cause the packet to be dropped if the flow identifier is not included in the flow table; and (2) cause the packet to be accepted for processing by the corresponding server if the flow identifier is included in the flow table.

### Claim 22 (depends on 21)

The multi-level packet distribution system of claim 21, wherein the packet examination module is operable to receive a second packet and operable to determine that the second packet is for a new flow by examining a header of the second packet, and further comprising:
a packet distribution algorithm operable to select a server to process the second packet, wherein the packet distribution algorithm is operable to select the server consistently with packet distribution algorithms of other distributed stateful second level packet distribution modules for the selected group;
a determination module operable to determine whether the server selected to process the second packet is the same as the server corresponding to the distributed stateful second level packet distribution module; and
a determination module operable to one of (1) cause the packet to be dropped if the selected server is not the same as the server corresponding to the distributed stateful second level packet distribution module; and (2) record a new flow identifier in a flow table of the packet distribution module and cause the packet to be accepted for processing by the selected server if the selected server is the same as the server corresponding to the distributed stateful second level packet distribution module.

### Claim 23 (depends on 20)

The multi-level packet distribution system of claim 20, wherein a distributed stateful second level packet distribution module corresponding to a server comprises:
a flow table that assigns flow identifiers to the server that corresponds to the distributed stateful second level packet distribution module;
a flow table checker module operable to determine whether a flow identifier corresponding to the flow for the packet is included in the flow table; and
an exists-on-others flow table, wherein the exists-on-others flow table includes flow identifiers assigned to servers of the selected group other than the server corresponding to the distributed stateful second level packet distribution module; and
a determination module operable to one of: (1) cause the packet to be accepted for processing by the corresponding server if the flow identifier is included in the flow table; and (2) cause the packet to be dropped if the flow identifier is not included in the flow table and the flow identifier is included in the exists-on-others flow table.

### Claim 24 (depends on 16)

The multi-level packet distribution system of claim 16, wherein the distributed stateful second level packet distribution system includes a plurality of distributed stateful second level packet distribution modules that each correspond to a different server of the selected group of servers, and wherein a distributed stateful second level packet distribution module corresponding to a server comprises:
a message send module operable to send a message to other distributed stateful second level packet distribution modules of the selected group, wherein the message indicates a state and a load of the corresponding server;
a message receive module operable to receive messages from the other distributed stateful second level packet distribution modules, wherein the received messages indicate states and loads of the corresponding servers; and
a packet distribution algorithm adjustment module operable to adjust a packet distribution algorithm used by the distributed stateful second level packet distribution module based on the states and loads indicated in the received messages.

### Claim 25 (depends on 16)

The multi-level packet distribution system of claim 16, wherein the distributed stateful second level packet distribution system includes at least one distributed stateful second level packet distribution module corresponding to the selected group of servers and at least one distributed stateful second level packet distribution module corresponding to a second group of servers, and wherein the at least one distributed stateful second level packet distribution module corresponding to the selected group of servers comprises:
a message receive module operable to receive a message at the at least one distributed stateful second level packet distribution module corresponding to the selected group of servers from the at least one distributed stateful second level packet distribution module corresponding to a second group of servers, wherein the received message indicates at least one load associated with the second group of servers; and
a server group re-assignment module operable to re-assign a server of the second group to the selected group based at least in part on the at least one load indicted in the received message and at least one load associated with the selected group of servers.
