Distributed packet deduplication

A computer-implemented method comprising:

• applying, by a network appliance, a specified transformation function to generate a value for each of a plurality of data packets received at a first ingress port of the network appliance, to produce a plurality of values;
• and
• using, by the network appliance, the plurality of values to identify
• a first batch of data packets to be examined by a local instance of a deduplication program that executes in the network appliance, and
• a second batch of data packets to be examined by a remote instance of the deduplication program that executes in another network appliance.

A network appliance comprising:

• an ingress port through which to receive virtualized traffic associated with a virtual machine;
• an egress port through which data packets are routed away from the network appliance;
• and
• a processor configured to
• generate a value for each data packet included in the virtualized traffic, thereby producing a plurality of values,
• identify a first value corresponding to a first data packet included in the virtualized traffic,
• access a data structure that includes an entry for each value generated by the processor,
• wherein each entry includes a routing instruction that specifies which instance of a plurality of instances of a deduplication program is responsible for examining the corresponding data packet;
• determine, based on the first value, that the first data packet is to be examined by a remote instance of the deduplication program that executes in another network appliance, and
• forward the first data packet to the egress port for transmission to the other network appliance.