# US 10,320,630 B2 · Hierarchichal sharding of flows from sensors to collectors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,320,630 B2
- Document ID: US10320630B2
- Status: Granted
- Kind: B2
- Application number: 15/171,855
- Filed: 2016-06-02
- Published: 2016-12-08
- Granted: 2019-06-11
- Earliest claimed: 2015-06-05
- Assignees: Cisco Technology, Inc.
- Applicants: Cisco Technology, Inc.
- Inventors: Shashidhar Gandham; Rohit Chandra Prasad; Abhishek Ranjan Singh; Navindra Yadav; Khawar Deen; Varun Sagar Malhotra
- Web page: https://patentgrep.com/patent/US10320630B2

## CPC

- G06F 9/45558: {Hypervisor-specific management and integration aspects}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04842: Selection of displayed objects or displayed text elements (G06F3/0482 takes precedence)
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 16/122: {using management policies (point-in-time backing up or restoration of persistent data G06F11/1446; file migration policies for HSM systems G06F16/185)}
- G06F 16/137: {Hash-based (content-based indexing of textual data G06F16/31)}
- G06F 16/162: {Delete operations (erasing in storage systems G06F3/0652)}
- G06F 16/17: Details of further file system functions
- G06F 16/173: {Customisation support for file systems, e.g. localisation, multi-language support, personalisation}
- G06F 16/174: Redundancy elimination performed by the file system (error detection or correction of the data by redundancy in operations G06F11/14)
- G06F 16/1744: {using compression, e.g. sparse files}
- G06F 16/1748: {De-duplication implemented within the file system, e.g. based on file segments (de-duplication techniques in storage systems for the management of data blocks G06F3/0641)}
- G06F 16/2322: {using timestamps}
- G06F 16/235: {Update request formulation}
- G06F 16/2365: {Ensuring data consistency and integrity}
- G06F 16/24578: {using ranking}
- G06F 16/248: Presentation of query results
- G06F 16/285: {Clustering or classification}
- G06F 16/288: {Entity relationship models}
- G06F 16/29: Geographical information databases
- G06F 16/9535: Search customisation based on user profiles and personalisation
- G06F 21/53: by executing in a restricted environment, e.g. sandbox or secure virtual machine
- G06F 21/552: {involving long-term monitoring or reporting}
- G06F 21/556: {involving covert channels, i.e. data leakage between processes (inhibiting the analysis of circuitry or operation with measures against power attack G06F21/755)}
- G06F 21/566: {Dynamic detection, i.e. detection performed at run-time, e.g. emulation, suspicious activities}
- G06N 20/00: Machine learning
- G06N 99/00: Subject matter not provided for in other groups of this subclass
- G06T 11/26: Drawing of charts or graphs
- H04J 3/0661: {using timestamps}
- H04J 3/14: Monitoring arrangements {(for SDH/SONET rings H04J3/085)}
- H04L 1/242: {by comparing a transmitted test signal with a locally generated replica}
- H04L 7/10: Arrangements for initial synchronisation
- H04L 9/0866: {involving user or device identifiers, e.g. serial number, physical or biometrical information, DNA, hand-signature or measurable physical characteristics}
- H04L 9/3239: {involving non-keyed hash functions, e.g. modification detection codes [MDCs], MD5, SHA or RIPEMD}
- H04L 9/3242: {involving keyed hash functions, e.g. message authentication codes [MACs], CBC-MAC or HMAC}
- H04L 41/046: comprising network management agents or mobile agents therefor
- H04L 41/0668: by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
- H04L 41/0803: Configuration setting
- H04L 41/0806: for initial configuration or provisioning, e.g. plug-and-play
- H04L 41/0816: the condition being an adaptation, e.g. in response to network events
- H04L 41/0893: Assignment of logical groups to network elements
- H04L 41/0894: Policy-based network configuration management
- H04L 41/12: Discovery or management of network topologies
- H04L 41/16: using machine learning or artificial intelligence
- H04L 41/22: comprising specially adapted graphical user interfaces [GUI]
- H04L 41/40: using virtualisation of network functions or resources, e.g. SDN or NFV entities
- H04L 43/02: Capturing of monitoring data
- H04L 43/026: using flow identification
- H04L 43/04: Processing captured monitoring data, e.g. for logfile generation
- H04L 43/045: for graphical visualisation of monitoring data
- H04L 43/062: related to network traffic
- H04L 43/08: Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L 43/0805: by checking availability
- H04L 43/0811: by checking connectivity
- H04L 43/0829: Packet loss
- H04L 43/0841: {Round trip packet loss}
- H04L 43/0858: {One way delays}
- H04L 43/0864: Round trip delays
- H04L 43/0876: Network utilisation, e.g. volume of load or congestion level
- H04L 43/0882: Utilisation of link capacity
- H04L 43/0888: Throughput
- H04L 43/10: Active monitoring, e.g. heartbeat, ping or trace-route
- H04L 43/106: using time related information in packets, e.g. by adding timestamps
- H04L 43/12: Network monitoring probes
- H04L 43/16: Threshold monitoring
- H04L 43/20: the monitoring system or the monitored elements being virtualised, abstracted or software-defined entities, e.g. SDN or NFV
- H04L 45/306: {Route determination based on the nature of the carried application}
- H04L 45/38: {Flow based routing}
- H04L 45/46: {Cluster building}
- H04L 45/507: {Label distribution}
- H04L 45/66: {Layer 2 routing, e.g. in Ethernet based MAN's}
- H04L 45/74: Address processing for routing
- H04L 47/11: Identifying congestion
- H04L 47/20: Traffic policing
- H04L 47/2441: relying on flow classification, e.g. using integrated services [IntServ]
- H04L 47/2483: involving identification of individual flows
- H04L 47/28: in relation to timing considerations
- H04L 47/31: by tagging of packets, e.g. using discard eligibility [DE] bits
- H04L 47/32: by discarding or delaying data units, e.g. packets or frames
- H04L 61/5007: Internet protocol [IP] addresses
- H04L 63/0227: {Filtering policies (mail message filtering H04L51/212)}
- H04L 63/0263: {Rule management}
- H04L 63/06: {for supporting key management in a packet data network (cryptographic mechanisms or cryptographic arrangements for key management H04L9/08)}
- H04L 63/0876: {based on the identity of the terminal or configuration, e.g. MAC address, hardware or software configuration or device fingerprint}
- H04L 63/1408: {by monitoring network traffic (monitoring network traffic per se H04L43/00)}
- H04L 63/1416: {Event detection, e.g. attack signature detection}
- H04L 63/1425: {Traffic logging, e.g. anomaly detection}
- H04L 63/1433: {Vulnerability analysis}
- H04L 63/1441: {Countermeasures against malicious traffic (countermeasures against attacks on cryptographic mechanisms H04L9/002)}
- H04L 63/145: {the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms}
- H04L 63/1458: {Denial of Service}
- H04L 63/1466: {Active attacks involving interception, injection, modification, spoofing of data unit addresses, e.g. hijacking, packet injection or TCP sequence number attacks}
- H04L 63/16: {Implementing security features at a particular protocol layer}
- H04L 63/20: {for managing network security; network security policies in general (filtering policies H04L63/0227)}
- H04L 67/01: Protocols
- H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46)
- H04L 67/1001: for accessing one among a plurality of replicated servers
- H04L 67/12: specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
- H04L 67/51: Discovery or management thereof, e.g. service location protocol [SLP] or web services
- H04L 67/75: Indicating network or usage conditions on the user display
- H04L 69/16: Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L 69/22: Parsing or analysis of headers
- H04W 72/54: based on quality criteria
- H04W 84/18: Self-organising networks, e.g. ad-hoc networks or sensor networks
- G06F 2009/4557: {Distribution of virtual machine instances; Migration and load balancing}
- G06F 2009/45587: {Isolation or security of virtual machine instances}
- G06F 2009/45591: {Monitoring or debugging support}
- G06F 2009/45595: {Network integration; Enabling network access in virtual machine instances}
- G06F 2221/033: Test or assess software
- G06F 2221/2101: Auditing as a secondary aspect
- G06F 2221/2105: Dual mode as a secondary aspect
- G06F 2221/2111: Location-sensitive, e.g. geographical location, GPS
- G06F 2221/2115: Third party
- G06F 2221/2145: Inheriting rights or properties, e.g., propagation of permissions or restrictions within a hierarchy
- H04L 67/535: {Tracking the activity of the user (network monitoring arrangements H04L43/00; recording of computer activity G06F11/34)}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04J: MULTIPLEX COMMUNICATION (transmission in general H04B; peculiar to transmission of digital information H04L5/00; systems for the simultaneous or sequential transmission of more than one television signal H04N7/08; in exchanges H04Q11/00; stereophonic systems H04S)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

Systems, methods, and computer-readable media for hierarchichal sharding of flows from sensors to collectors. A first collector can receive a first portion of a network flow from a first capturing agent and determine that a second portion of the network flow was not received from the first capturing agent. The first collector can then send the first portion of the network flow to a second collector. A third collector can receive the second portion of the network flow from a second capturing agent and determine that the third collector did not receive the first portion of the network flow. The third collector can then send the second portion of the network flow to the second collector. The second collector can then aggregate the first portion and second portion of the network flow to yield the entire portion of the network flow.

## Claims

### Claim 1

A method for managing a plurality of collector devices including at least first, second and third collector devices, the method comprising:
receiving, by the first collector device, a first portion of a network flow from a first capturing agent, the first portion of the network flow being captured by the first capturing agent at a first host associated with the first capturing agent;
determining, by the first collector device, that a second portion of the network flow was not received, to yield a first determination;
based on the first determination, sending, by the first collector device, the first portion of the network flow to the second collector device;
receiving, by the third collector device, the second portion of the network flow from a second capturing agent, the second portion of the network flow being captured by the second capturing agent at a second host associated with the second capturing agent;
determining, by the third collector device, that the first portion of the network flow was not received, to yield a second determination;
based on the second determination, sending, by the third collector device, the second portion of the network flow to the second collector device;
determining, by the second collector device, that the first portion of the network flow and the second portion of the network flow are part of a same network flow; and
aggregating, by the second collector device, the first portion of the network flow and the second portion of the network flow to yield an aggregated network flow.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collector devices in the respective shards, wherein each of the respective shards comprises a number of assigned collector devices from the plurality of collector devices.

### Claim 3 (depends on 2)

The method of claim 2, wherein the first portion of the network flow is transmitted by the first capturing agent to a first shard from the respective shards, and the second portion of the network flow is transmitted by the second capturing agent to a second shard from the respective shards, wherein the first collector device is part of the first shard and the third collector device is part of the second shard.

### Claim 4 (depends on 2)

The method of claim 2, wherein the second collector device is part of a shard from the respective shards, and wherein the second collector device is assigned a flow key and hash that corresponds to the network flow.

### Claim 5 (depends on 4)

The method of claim 4, wherein the second collector device is selected to receive the first portion and the second portion of the network flow from a plurality of other collector devices in the shard based on the flow key and hash corresponding to the network flow.

### Claim 6 (depends on 1)

The method of claim 1, wherein the first collector device and the third collector device are mapped to respective shards from a first layer of shards, each of the respective shards comprising a selected group of collector devices, and wherein the second collector device is mapped to a shard from a second layer of shards.

### Claim 7 (depends on 6)

The method of claim 6, wherein the first layer of shards is designated to receive network flow data directly from capturing agents deployed throughout a network, and wherein the second layer of shards is designated to receive different portions of the network flow data directly from different collector devices in the first layer of shards.

### Claim 8 (depends on 1)

The method of claim 1, wherein the first capturing agent comprises at least one of a process, a kernel module, or a software driver.

### Claim 9 (depends on 8)

The method of claim 8, wherein the first capturing agent resides in one of a hypervisor on the host or a virtual machine on the hypervisor, the network flow being sent or received by the one of the hypervisor or the virtual machine.

### Claim 10

A system for managing a plurality of collector devices including at least first, second and third collector devices, the system comprising:
one or more processors; and
one or more computer-readable storage devices having stored therein instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving, by the first collector device, a first portion of a network flow from a first capturing agent, the first portion of the network flow being captured by the first capturing agent at a first host associated with the first capturing agent;
determining, by the first collector device, that a second portion of the network flow was not received, to yield a first determination;
based on the first determination, sending, by the first collector device, the first portion of the network flow to the second collector device;
receiving, by the third collector device, the second portion of the network flow from a second capturing agent, the second portion of the network flow being captured by the second capturing agent at a second host associated with the second capturing agent;
determining, by the third collector device, that the first portion of the network flow was not received, to yield a second determination;
based on the second determination, sending, by the third collector device, the second portion of the network flow to the second collector device;
determining, by the second collector device, that the first portion of the network flow and the second portion of the network flow are part of a same network flow; and
aggregating, by the second collector device, the first portion of the network flow and the second portion of the network flow to yield an aggregated network flow.

### Claim 11 (depends on 10)

The system of claim 10, the one or more computer-readable storage devices storing additional instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collector devices in the respective shards, wherein each of the respective shards comprises a number of assigned collector devices from the plurality of collector devices.

### Claim 12 (depends on 11)

The system of claim 11, wherein the first portion of the network flow is transmitted by the first capturing agent to a first shard from the respective shards, and the second portion of the network flow is transmitted by the second capturing agent to a second shard from the respective shards, wherein the first collector device is part of the first shard and the third collector device is part of the second shard.

### Claim 13 (depends on 11)

The system of claim 11, wherein the second collector device is part of a shard from the respective shards, and wherein the second collector device is assigned a flow key and hash that corresponds to the network flow.

### Claim 14 (depends on 13)

The system of claim 13, wherein the second collector device is selected to receive the first portion and the second portion of the network flow from a plurality of other collector devices in the shard based on the flow key and hash corresponding to the network flow.

### Claim 15

A non-transitory computer-readable storage device storing instructions for managing a plurality of collector devices including at least first, second and third collector devices, which, when executed by a processor, cause the processor to perform operations comprising:
receiving, by the first collector device, a first portion of a network flow from a first capturing agent, the first portion of the network flow being captured by the first capturing agent at a first host associated with the first capturing agent;
determining, by the first collector device, that a second portion of the network flow was not received, to yield a first determination;
based on the first determination, sending, by the first collector device, the first portion of the network flow to the second collector device;
receiving, by the third collector device, the second portion of the network flow from a second capturing agent, the second portion of the network flow being captured by the second capturing agent at a second host associated with the second capturing agent;
determining, by the third collector device, that the first portion of the network flow was not received, to yield a second determination;
based on the second determination, sending, by the third collector device, the second portion of the network flow to the second collector device;
determining, by the second collector device, that the first portion of the network flow and the second portion of the network flow are part of a same network flow; and
aggregating, by the second collector device, the first portion of the network flow and the second portion of the network flow to yield an aggregated network flow.

### Claim 16 (depends on 15)

The computer-readable storage device of claim 15, storing additional instructions which, when executed by the processor, cause the processor to perform operations comprising:
assigning a plurality of capturing agents deployed throughout a network to respective shards, each of the plurality of capturing agents being configured to capture network activity associated with a respective host and report the network activity to one or more collector devices in the respective shards, wherein each of the respective shards comprises a number of assigned collector devices from the plurality of collector devices.

### Claim 17 (depends on 16)

The computer-readable storage device of claim 16, wherein the first portion of the network flow is transmitted by the first capturing agent to a first shard from the respective shards, and the second portion of the network flow is transmitted by the second capturing agent to a second shard from the respective shards, wherein the first collector device is part of the first shard and the third collector device is part of the second shard.

### Claim 18 (depends on 16)

The computer-readable storage device of claim 16, wherein:
the second collector device is part of a shard from the respective shards;
the second collector device is assigned a flow key and hash that corresponds to the network flow; and
the second collector device is selected to receive the first portion and the second portion of the network flow from a plurality of other collector devices in the shard based on the flow key and hash corresponding to the network flow.

### Claim 19 (depends on 15)

The computer-readable storage device of claim 15, wherein:
the first collector device and the third collector device are mapped to respective shards from a first layer of shards, each of the respective shards comprising a selected group of collector devices, and wherein the second collector device is mapped to a shard from a second layer of shards; and
the first layer of shards is designated to receive network flow data directly from capturing agents deployed throughout a network, and wherein the second layer of shards is designated to receive different portions of the network flow data directly from different collector devices in the first layer of shards.