Methods for intelligent load balancing and high speed intelligent network recorders

A high speed intelligent network recording system for recording a plurality of flows of network packets into a computer network, the high speed intelligent network recording system comprising:

• a controller unit having
• an enclosure,
• a backplane printed board (PCB) mounted in the enclosure, and
• a plurality of controller cards mounted in the enclosure coupled to the backplane PCB;
• and
• a storage unit coupled in communication to the controller unit, the storage unit including
• an enclosure,
• a backplane printed board (PCB) mounted in the enclosure,
• a plurality of drive trays mounted in the enclosure coupled to the backplane PCB, and
• a plurality of pluggable storage drives coupled to each of the plurality of drive trays;
• wherein each controller card in the controller unit includes a first microcomputer coupled in communication with a plurality of storage devices in the storage unit.

A method for recording and analytics of a plurality of packet records for network data packets entering a network over a relevant data time window, the method comprising:

• receiving a plurality of incoming packets including ingress packets and egress packets, wherein each incoming packet belongs to one of a plurality of conversation flows;
• forming a capture stream of packet records for the incoming packets, wherein each packet record includes metadata having a timestamp, a record length, and a flow hash;
• and
• performing intelligent load balancing on the capture stream of packet records, the load balancing including
• reading the metadata for each packet record,
• determining a packet record is part of either a hot flow or a cold flow,
• selecting a destination node for each packet record based on the flow hash, and
• steering the packet record to one of a plurality of encapsulation buffers based on the destination node, wherein a cold flow tends to be directed coherently to a single node.

A computer-readable product for recording and analytics of a plurality of packet records for network data packets entering a network over a relevant data time window, the computer-readable product including a non-transitory computer-readable storage medium storing instructions that when executed perform the functions comprising:

• receiving a plurality of incoming packets including ingress packets and egress packets, wherein each incoming packet belongs to one of a plurality of conversation flows;
• forming a capture stream of packet records for the incoming packets, wherein each packet record includes metadata having a timestamp, a record length, and a flow hash;
• and
• performing intelligent load balancing on the capture stream of packet records, the load balancing including
• reading the metadata for each packet record,
• determining a packet record is part of either a hot flow or a cold flow,
• selecting a destination node for each packet record based on the flow hash, and
• steering the packet record to one of a plurality of encapsulation buffers based on the destination node, directed coherently to a single node.

A method for a query return process for a network recorder on a network, the method comprising:

• receiving a query from a query requester;
• multicasting the query to a plurality of nodes, wherein a plurality of incoming packets to the network have undergone cold balancing among the plurality of nodes;
• receiving query responses from the plurality of nodes;
• transferring the query responses to a plurality of query response buffers;
• performing a time-ordered merge by inspecting timestamps of un-merged query responses from the query response buffers;
• and
• merging the un-merged query responses into a time-order response stream for response to the query requester.

A computer-readable product for a query return process for a network recorder on a network, the computer-readable product including a non-transitory computer-readable storage medium storing instructions that when executed perform the functions comprising:

• receiving a query from a query requester;
• multicasting the query to a plurality of nodes, wherein a plurality of incoming packets to the network have undergone cold balancing among the plurality of nodes;
• receiving query responses from the plurality of nodes;
• transferring the query responses to a plurality of query response buffers;
• performing a time-ordered merge by inspecting timestamps of un-merged query responses from the query response buffers;
• and
• merging the un-merged query responses into a time-order response stream for response to the query requester.