# US 2015/0103661 A1 · Host side method of using a controller assignment list

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0103661 A1
- Document ID: US20150103661A1
- Status: Published application
- Kind: A1
- Application number: 14/070,451
- Filed: 2013-11-01
- Published: 2015-04-16
- Earliest claimed: 2013-10-10
- Assignees: VMware, Inc.
- Applicants: VMware, Inc.
- Inventors: Jianjun Shen; Hua Wang; James Joseph Stabile; Xuan Zhang
- Web page: https://patentgrep.com/patent/US20150103661A1

## CPC

- H04L 41/042: comprising distributed management centres cooperatively managing the network
- G06F 9/452: {Remote windowing, e.g. X-Window System, desktop virtualisation (protocols for virtual reality H04L67/131)}
- H04L 41/344: Out-of-band transfers
- H04L 45/64: using an overlay routing layer
- H04L 47/125: by balancing the load, e.g. traffic engineering
- H04L 67/131: Protocols for games, networked simulations or virtual reality
- H04L 41/0659: by isolating or reconfiguring faulty entities
- H04L 61/103: across network layers, e.g. resolution of network layer into physical layer addresses or address resolution protocol [ARP]
- H04L 61/5014: using dynamic host configuration protocol [DHCP] or bootstrap protocol [BOOTP]
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Some embodiments provide a novel method for distributing control-channel communication load between multiple controllers in a network control system. In some embodiments, the controllers manage physical forwarding elements that forward data between several computing devices (also called hosts or host computers), some or all of which execute one or more virtual machines (VMs). The method of some embodiments distributes a controller assignment list to the host computers. The host computers use this list to identify the controllers with which they need to interact to perform some of the forwarding operations of their associated logical forwarding elements. In some embodiments, agents executing on the host computers (1) review the controller assignment list to identify the appropriate controllers, and (2) establish control channel communications with these controllers to obtain the needed data for effectuating the forwarding operations of their associated physical forwarding elements. These agents in some embodiments are responsible for out-of-band control channel communication with the controllers.

## Claims

### Claim 1

For a computer that executes at least one physical forwarding element that together with physical forwarding elements of other computers implements a plurality different types of logical forwarding elements (LFEs), a method of identifying the controllers for managing the LFEs, the method comprising:
receiving a controller assignment list that, based on a common range of index values, specifies N controllers for N sets of LFEs;
for a first LFE of a first LFE type, generating a first index value based on an identifier of the first LFE and using the first index value to identify a first controller in the controller assignment list as the first LFE's controller; and
for a second LFE of a second LFE type, generating a second index value based on an identifier of the second LFE and using the second index value to identify a second controller in the controller assignment list as the second LFE's controller.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating either the first index value or the second index values comprises using a hashing operation that includes a modulo operation to convert an identifier of an LFE to an index value that falls within the common range of index values.

### Claim 3 (depends on 2)

The method of claim 2, wherein the modulo operation has a range of remainder values that coincides with the common range of index values.

### Claim 4 (depends on 1)

The method of claim 1, wherein the first LFE is a logical switch and the second LFE is a logical router, and wherein the first LFE's identifier is a logical switch identifier and the second LFE's identifier is a logical router identifier.

### Claim 5 (depends on 1)

The method of claim 1 further comprising:
after identifying the first controller, establishing a control channel connection with the first controller;
using the control channel with the first controller to obtain data to effectuate a subset of operations for the first LFE.

### Claim 6 (depends on 5)

The method of claim 5 further comprising:
in a data storage, storing a set of parameters that identify the control channel connection with the first controller;
receiving a particular operation for the first LFE;
generating a data request to a controller for data to facilitate the processing of the received particular first LFE operation;
from the data storage, retrieving the set of parameters that identify the first controller as the controller to process the data request;
wherein using the control channel comprises sending the data request to the first controller in order to obtain data necessary to effectuate the processing of the particular first LFE operation.

### Claim 7 (depends on 6)

The method of claim 6, wherein the set of parameters comprises the first index value for the first LFE and an identifier identifying the first controller, the method further comprising:
generating the first index value from the first LFE's identifier;
using the generated first index value to retrieve the identifier of the first controller from the data storage.

### Claim 8 (depends on 7)

The method of claim 7, wherein the stored identifier for the first controller is a network address of the first controller.

### Claim 9 (depends on 8)

The method of claim 8, wherein the network address is an Internet Protocol (IP) address.

### Claim 10

A host computer comprising:
a first physical forwarding element (PFE) that collectively with physical forwarding elements of other host computers implements a plurality logical forwarding elements (LFEs);
a plurality of virtual machines (VMs) with each VM associated with at least one LFE;
at least one proxy for converting a subset of packets that the VMs send along the datapath into an out-of-band control channel messages to a plurality of controllers responsible for managing the LFEs associated with the VMs;
at least one agent for examining a controller assignment list to identify the controller for each LFE that should receive the control channel messages for that LFE, wherein the controller assignment list distributes the control channel processing for the N sets of LFEs among the controllers in a manner to distribute the control channel load on the plurality of controllers.

### Claim 11 (depends on 10)

The host of claim 10, wherein
the controller assignment list specifies the controllers for the N sets of LFEs based on the index values;
the agent is further for receiving the controller assignment list from one of the controllers; and
the agent identifies the controllers for the LFE by generating the index values from the identifiers of the LFEs, and using the generated index values to retrieve from the controller assignment list the identity of the controllers for the LFEs.

### Claim 12 (depends on 11)

The host of claim 11, wherein the agent is further for:
receiving the controller assignment list from one of the controllers;
establishing a control channel connection with each controller in the controller assignment list that is specified as the controller for one of the LFEs implemented by the first PFE; and
storing in a connection list information about each controller with which it established a control channel connection in order to use the stored information subsequently to identify the controllers to send subsequent control channel messages.

### Claim 13 (depends on 10)

The host of claim 10, wherein the agent is further for subscribing to one of the controllers to receive the controller assignment list and updates to this list.

### Claim 14 (depends on 10)

The host of claim 10, wherein a master controller generates the controller assignment list, wherein the controller to which the agent subscribes does not have to be the master controller.

### Claim 15 (depends on 10)

The host of claim 10, wherein the controller assignment list is generated by a process that distribute the responsibility for the LFEs based on a set of criteria that includes a uniform distribution of the LFEs among the controllers, such that each controller has approximately the same number of LFEs to manage as other controllers.

### Claim 16 (depends on 10)

The host of claim 10, wherein the packets from the VMs are in the logical domain and are intended for processing by the LFEs of the VMs, wherein the proxy is further for intercepting the packets from the VMs along the datapath to the particular PFE.

### Claim 17 (depends on 10)

The host of claim 10 further comprising one proxy for each VM, wherein each VM's proxy examines the packets sent from the VM to determine whether the packet should be processed by out-of-band control channel data.

### Claim 18 (depends on 10)

The host of claim 10, wherein
the LFEs are a first type of LFEs,
the host further comprises a second PFE that collectively with physical forwarding elements of other host computers implements a plurality logical forwarding elements (LFEs) of a second type;
the controller assignment list uses a common range of index values to identify the controllers;
the agent is further for using the identifiers of the first and second type LFEs to derive index values that identify the controllers for the LFEs in the controller assignment list; and
the same index value is derivable from the first and second identifiers for a first-type LFE and a second-type LFE, and the same index value identifies the same controller for both LFEs in the controller assignment list.

### Claim 19 (depends on 18)

The host of claim 18, wherein the first-type LFE and the second-type LFE are implemented by the first and second PFEs, respectively.

### Claim 20 (depends on 18)

The host of claim 18, wherein the first-type LFE is implemented by the first PFE, but the second-type LFE is not implemented by the second PFE.

### Claim 21 (depends on 10)

The host of claim 10, wherein:
the LFEs are a first type of LFEs,
the host further comprises a second PFE that collectively with physical forwarding elements of other host computers implements a plurality logical forwarding elements (LFEs) of a second type;
the controller assignment list uses a common range of index values to identify the controllers;
the agent is further for using the identifiers of the first and second type LFEs to generates index values that identify the controllers for the LFEs in the controller assignment list; and
the agent generates a first and second index values from the first and second identifiers for a first-type LFE and a second-type LFE implemented by the first and second PFEs, wherein the first index value identifies a first controller and the second index value identifies a second, different controller in the controller assignment list.

### Claim 22

A machine readable medium storing a program for a host computer that executing a plurality of virtual machines (VMs) and a physical forwarding element (PFE) that collectively with PFEs of other host computers implements a plurality logical forwarding elements (LFEs), the program comprising sets of instructions for:
receiving a controller assignment list that specifies controllers for a plurality of index values;
for at least two LFEs, generating two index values based on the identifiers of the two LFEs;
using the index values to identify in the controller assignment list two controllers that are associated with the two LFEs;
establishing two control channel connections with the two controllers;
using the control channel connection with the each particular controller to obtain data to effectuate data processing for the LFE associated with the particular controller.

### Claim 23 (depends on 22)

The machine readable medium of claim 22, wherein the two LFEs are first and second LFEs, and the two control channel connections are first and second control channel connections, the program further comprising sets of instructions for:
in a data storage, storing an identity of each controller with an established control channel connection;
receiving a particular operation for the first LFE;
generating the first index value from the first LFE's identifier;
using the generated first index value to retrieve the first controller's identity from the data storage;
wherein the set of instructions for using the first control channel comprises a set of instructions for using the first control channel to send a data request to the first controller after retrieving the first controller's identity from the data storage, in order to obtain data necessary to effectuate the processing of the particular LFE operation.

### Claim 24 (depends on 23)

The machine readable medium of claim 23, wherein the stored identity of the first controller is a network address of the first controller.

### Claim 25 (depends on 24)

The machine readable medium of claim 24, wherein the network address is an Internet Protocol (IP) address.
