Grouping network traffic prior to storage in a columnar database

A computer-implemented method of grouping network traffic metadata, the method comprising:

• receiving network traffic metadata from a plurality of network routers;
• obtaining a statistic about flow of network traffic metadata received for a property range per router of the plurality of routers;
• distributing the network traffic metadata into a plurality of groups in order for network traffic metadata from the smallest possible number of routers to be distributed to each group, with the statistic about flow of network traffic metadata distributed optimally for a criteria regarding the statistic amongst the plurality of groups for minimizing cardinality of each group of the plurality of groups with respect to different dimensions of the network traffic metadata;
• providing each group to a columnar database for storage of the network traffic metadata distributed into each group in a different partition of the columnar database;
• determining an ideal weight as a function of a sum of the statistic about flow of network traffic metadata for all of the instances of a selected dimension and a number of groups included in the plurality of groups;
• splitting each router into equally sized chunks, wherein the number of chunks is a function of the statistic for the router and the ideal weight;
• iteratively repeating until a total number of chunks for all of the instances of the selected dimension exceeds the number of groups;
• and
• splitting the router having the largest size chunk or chunks into its current number of chunks incremented by one.

A system of grouping network traffic metadata, the system comprising:

• a memory configured to store a plurality of programmable instructions;
• and
• least one processing device in communication with the memory, wherein the at least one processing device, upon execution of the plurality of programmable instructions is configures to:
• based on a selected dimension of the network traffic metadata received from at least one network router, obtaining a statistic about a flow of network traffic metadata received over an interval for each instance of the selected dimension over a property range;
• distributing the network traffic metadata into a plurality of groups for network traffic metadata from the smallest possible number of instances of the selected dimension to be distributed to each group, with the flow of network traffic metadata distributed optimally for a criteria regarding the statistic amongst the plurality of groups for minimizing cardinality of each group of the plurality of groups with respect to unselected dimensions of the network traffic metadata;
• and
• providing each group to a columnar database for storage of the network traffic metadata distributed into each group in a different partition of the columnar database;
• determining an ideal weight as a function of a sum of the statistic about flow of network traffic metadata for all of the instances of the selected dimension and a number of groups included in the plurality of groups;
• splitting each router into equally sized chunks, wherein the number of chunks is a function of the statistic for the router and the ideal weight;
• iteratively repeating until a total number of chunks for all of the instances of the selected dimension exceeds the number of groups;
• and
• splitting the router having the largest size chunk or chunks into its current number of chunks incremented by one.