# US 2019/0379588 A1 · Distributed packet deduplication

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0379588 A1
- Document ID: US20190379588A1
- Status: Published application
- Kind: A1
- Application number: 16/001,721
- Filed: 2018-06-06
- Published: 2019-12-12
- Earliest claimed: 2018-06-06
- Assignees: Gigamon Inc.
- Applicants: Gigamon Inc.
- Inventors: Anil RAO
- Web page: https://patentgrep.com/patent/US20190379588A1

## CPC

- H04L 43/028: by filtering
- H04L 47/125: by balancing the load, e.g. traffic engineering
- H04L 45/38: {Flow based routing}
- H04L 47/32: by discarding or delaying data units, e.g. packets or frames
- H04L 67/1023: based on a hash applied to IP addresses or costs
- H04L 67/1034: Reaction to server failures by a load balancer
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

Introduced here are network visibility appliances capable of implementing a distributed deduplication scheme by routing traffic amongst multiple instances of a deduplication program. Data traffic can be forwarded to a pool of multiple network visibility appliances that collectively ensure no duplicate copies of data packets exist in the data traffic. The network visibility appliances can route the traffic to different instances of the deduplication program so that duplicate copies of a data packet are guaranteed to arrive at the same instance of the deduplication program, regardless of which network visibility appliance(s) initially received the duplicate copies of the data packet.

## Claims

### Claim 1

A computer-implemented method comprising:
receiving virtualized traffic at a first ingress port of a first network visibility appliance;
separating the virtualized traffic into
a first batch of data packets that share a first characteristic in common, the first batch to be examined by a first instance of a deduplication program that executes in the first network visibility appliance, and
a second batch of data packets that share a second characteristic in common, the second batch to be examined by a second instance of the deduplication program that executes in a second network visibility appliance;
receiving a third batch of data packets at a second ingress port of the first network visibility appliance;
determining that all data packets in the third batch share the first characteristic in common;
forwarding all data packets of the first batch and all data packets of the third batch to the first instance of the deduplication for examination; and
forwarding all data packets of the second batch to an egress port for transmission to the second network visibility appliance.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the virtualized traffic is associated with a first virtual machine, and wherein the third batch of data packets is included in virtualized traffic associated with a second virtual machine.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein said separating comprises:
generating a hash value for each data packet included in the virtualized traffic, thereby producing a plurality of hash values;
accessing a data structure that maps the plurality of hash values amongst a plurality of instances of the deduplication program,
wherein each hash value is mapped to only a single instance of the deduplication program, and
wherein each instance of the deduplication program executes in a different network visibility appliance;
determining that each data packet in the first batch of data packets corresponds to an entry in the data structure that specifies the first instance of the deduplication program; and
determining that each data packet in the second batch of data packets corresponds to an entry in the data structure that specifies the second instance of the deduplication program.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, further comprising:
receiving an indication that the second network visibility appliance is not presently accessible; and
modifying entries in the data structure that correspond to the second batch of data packets to indicate a third instance of the deduplication program,
wherein the third instance of the deduplication program resides on a third network visibility appliance, and
wherein said modifying causes the second batch of data packets to be forwarded to a second egress port for transmission to the third network visibility appliance.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the virtualized traffic is associated with a plurality of virtual machines.

### Claim 6

A computer-implemented method comprising:
applying, by a network appliance, a specified transformation function to generate a value for each of a plurality of data packets received at a first ingress port of the network appliance, to produce a plurality of values; and
using, by the network appliance, the plurality of values to identify
a first batch of data packets to be examined by a local instance of a deduplication program that executes in the network appliance, and
a second batch of data packets to be examined by a remote instance of the deduplication program that executes in another network appliance.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, further comprising:
causing, by the network appliance, the first batch of data packets to be examined by the local instance of the deduplication program; and
forwarding, by the network appliance, at least a portion of the first batch of data packets to a tool port for transmission to a network tool.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, further comprising:
forwarding, by the network appliance, the second batch of data packets to an egress port for transmission to the other network appliance.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, further comprising:
receiving, by the network appliance, a third batch of data packets from the other network appliance at a second ingress port;
causing, by the network appliance, the third batch of data packets to be examined by the local instance of the deduplication program; and
forwarding, by the network appliance, at least a portion of the third batch of data packets to the tool port for transmission to the network tool.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9, wherein the second ingress port corresponds to one end of a tunnel connected between the network appliance and the other network appliance.

### Claim 11 (depends on 6)

The computer-implemented method of claim 6, wherein using the plurality of values to identify the first and second batches of data packets comprises:
accessing, by the network appliance, a data structure that maps the plurality of values amongst a plurality of instances of the deduplication program,
wherein each value is mapped to only a single instance of the deduplication program, and
wherein each instance of the deduplication program executes in a different network appliance;
determining, by the network appliance, that each data packet in the first batch of data packets corresponds to an entry that specifies the local instance of the deduplication program; and
determining, by the network appliance, that each data packet in the second batch of data packets corresponds to an entry that specifies the remote instance of the deduplication program.

### Claim 12 (depends on 6)

The computer-implemented method of claim 6, wherein each value is based on a field in a header of the corresponding data packet.

### Claim 13 (depends on 6)

The computer-implemented method of claim 6, wherein each value is based on at least one of: Transmission Control Protocol (TCP) sequence number, header length, payload length, type of service, protocol, source address, or destination address.

### Claim 14

A network appliance comprising:
an ingress port through which to receive virtualized traffic associated with a virtual machine;
an egress port through which data packets are routed away from the network appliance; and
a processor configured to
generate a value for each data packet included in the virtualized traffic, thereby producing a plurality of values,
identify a first value corresponding to a first data packet included in the virtualized traffic,
access a data structure that includes an entry for each value generated by the processor,
wherein each entry includes a routing instruction that specifies which instance of a plurality of instances of a deduplication program is responsible for examining the corresponding data packet;
determine, based on the first value, that the first data packet is to be examined by a remote instance of the deduplication program that executes in another network appliance, and
forward the first data packet to the egress port for transmission to the other network appliance.

### Claim 15 (depends on 14)

The network appliance of claim 14, wherein the processor is further configured to:
identify a second value corresponding to a second data packet included in the virtualized traffic;
determine, based on the second value, that the second data packet is to be examined by a local instance of the deduplication program that executes in the network appliance;
cause the second data packet to be examined by the local instance of the deduplication program.

### Claim 16 (depends on 15)

The network appliance of claim 15, further comprising:
a tool port through which to route at least some of the virtualized traffic to a network tool.

### Claim 17 (depends on 16)

The network appliance of claim 16, wherein the processor is further configured to:
monitor whether the second data packet is filtered by the local instance of the deduplication program; and
responsive to a determination that the second data packet was not filtered by the local instance of the deduplication program,
forward the second data packet to the tool port for transmission to the network tool.

### Claim 18 (depends on 16)

The network appliance of claim 16, wherein the tool port corresponds to one end of a tunnel connected between the network appliance and the network tool.

### Claim 19 (depends on 14)

The network appliance of claim 14, wherein each instance of the plurality of instances of the deduplication program executes in a different network appliance within a visibility fabric.

### Claim 20 (depends on 14)

The network appliance of claim 14,
wherein the ingress port corresponds to one end of a first tunnel connected between the network appliance and the virtual machine, and
wherein the egress port corresponds to one end of a second tunnel connected between the network appliance and the other network appliance.
