Method and system for monitoring and analysis of network traffic flows

A system for processing traffic flow data received from a network probe device, comprising:

• a processor and a non-transitory computer readable storage medium having computer readable instructions stored thereon for execution by the processor, causing the processor to:
• (a) determine a statistical traffic flow profile of traffic flow data, comprising a distribution of number of traffic flows according to respective byte sizes;
• (b) select, from said distribution, a given byte size so that an aggregate of bytes in traffic flows having a size below the given byte size represents a predetermined small percentage of aggregate bytes in all traffic flows in said distribution;
• and
• (c) retain flow records of those traffic flows whose byte size is above the given byte size, and discard flow records for other traffic flows;
• retaining the flow records in memory buckets of a first and a second bucket chains, corresponding to a first and second predetermined time windows, and aggregating bytes sizes of flows in each memory bucket;
• wherein the first predetermined time window is T seconds, and the second predetermined time window is equal to, and coincident with a predetermined number of the first predetermined time windows;
• and
• splitting a flow record characterizing a traffic flow, which extends over parts of two or more consecutive predetermined time windows, into two or more flow records, each having partial traffic flow sizes assigned to equal a respective fraction of the traffic flow size according to a duration of the traffic flow that falls in each corresponding predetermined time window respectively.