US 9,426,071 B1 · Storing network bidirectional flow data and metadata with efficient processing technique

Abstract

A processing technique provides an improved indexing arrangement that enables storage, filtering and querying of metadata used to retrieve packets captured from a network and persistently stored in a data repository. A packet capture engine records the packets in packet capture (PCAP) formats from a network link at a substantially high packet transfer rate to persistent storage of the data repository in a sustained manner. Efficient filtering and querying of the metadata to retrieve the stored packets may be achieved, in part, by organizing the metadata as one or more metadata repositories. The processing technique uses the Berkeley Packet Filter (BPF) language as an interface of a BPF engine to search or index the stored packets in response to queries. The BPF engine processes BPF expressions used as precursors to the indexing arrangement to enable access to the repositories when searching and locating stored packets matching the expressions.

Claims20

Dependent claims

A method comprising:

intercepting a first packet from a network link at a node coupled to the network link, the node including persistent storage devices organized as a plurality of volumes having a hierarchical file system;
computing a first hash value based on a network flow of the first packet;
recording the first packet in a packet capture (PCAP) format as a first PCAP record including the first hash value and first flow metadata appended to the first PCAP record;
copying the first PCAP record to a first metadata repository stored as a first file on a first volume of the hierarchical file system of the node, wherein the first metadata repository stores a plurality of second PCAP records having second hash values and second flow metadata;
and
concurrently searching and retrieving one or more of the second PCAP records of the first metadata repository while copying the first packet to a data repository stored as a second file of a second volume of the hierarchical file system of the node to realize a substantially high sustained packet transfer rate of the network link.

3 more elements

A system comprising:

one or more processors of a node coupled to a network link, the node including persistent storage devices organized as a plurality of volumes having a hierarchical file system;
a plurality of storage repositories organized as an indexing arrangement and coupled to the one or more processors, the storage repositories including (i) a data repository configured to store packets intercepted from the network link, (ii) a first metadata repository configured to store flow records having hash values and flow metadata, and (iii) a second metadata repository configured to store entries having the hash values along with paths and offsets to the packets stored in the data repository;
and
a memory coupled to the one or more processors and configured to store one or more processes of an operating system, the one or more processes executable by the one or more processors to utilize the indexing arrangement to concurrently search and retrieve one or more of the flow records from the first metadata repository stored as a first file of a first volume of the hierarchical file system while copying one or more of the packets to the data repository for storage on a second file of a second volume of the hierarchical file system to realize a substantially high sustained packet transfer rate of the network link.

1 more element

Dependencies are inferred from the claim text.

Storing network bidirectional flow data and metadata with efficient processing technique

Life of this patent

Storing network bidirectional flow data and metadata with efficient processing technique

Life of this patent

Abstract

Claims20