Streaming method and system for processing network metadata

A method of processing network metadata generated on a network transmitting network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the method comprising the steps of:

• receiving network metadata from a plurality of sources in a data processing system, in at least one data format;
• determining the type or character of said network metadata;
• processing said network metadata by applying at least one policy governing network metadata processing, wherein said at least one policy includes the steps of:
• collecting information about internal network devices communicating with external network devices;
• said information comprising a list of communicating devices' IP addresses and protocols;
• applying a streaming cluster analysis method to determine patterns in communications between the communicating devices;
• identifying abnormal instances of communications between the communicating devices by the presence of small disjoint patterns in the set of all computed patterns;
• generating an alert message in a timely manner to inform of a potential botnet infection if at least one abnormal communication pattern is identified;
• and
• converting at least a portion of said network metadata into one or more different data formats that are used in said data processing system for other system metadata, in response, at least in part, to the results of said determining step.

A method of processing network metadata generated on a network transmitting network traffic using one or more network protocols, the network including devices at least some of which receive network traffic through an ingress interface and transmit network traffic through an egress interface, the method comprising the steps of:

• receiving network metadata from a plurality of sources in a data processing system, in at least one data format;
• determining the type or character of said network metadata;
• processing said network by applying at least one policy governing network metadata processing, wherein said at least one policy includes the steps of:
• comparing the source of incoming network traffic to a predefined list of monitored devices on said network;
• if the source IP address is on a predefined list of devices, storing the source IP in an in-memory database along with the number of bytes and packets reported in the ingress NetFlow record;
• at predefined intervals examining the in-memory database and identifying devices which exceeded a threshold set forth by a network manager;
• generating an alert message in a timely manner to inform of a traffic spike;
• and
• converting at least a portion of said network metadata into one or more different data formats that are used in said data processing system for other system metadata, in response, at least in part, to the results of said determining step.