Systems, methods and computer program products for scalable, low-latency processing of streaming data

A method of processing network flow monitoring data, the method comprising:

• receiving at least one incoming data stream of network monitoring messages;
• directing the network monitoring messages to a data processing segment comprising a load balancing operator, a data processing sequence that includes a plurality of collector processing sub-units and a plurality of parser sub-units, and a combiner operator, wherein the plurality of parser sub-units are directly connected to the plurality of collector processing sub-units, the plurality of parser sub-units are directly connected to the combiner operator, and the plurality of collector processing sub-units are directly connected to the load balancing operator such that data flows through the data processing segment sequentially from the load balancing operator to the plurality of collector processing sub-units to the plurality of parser sub-units to the combiner operator;
• distributing by the load balancing operator, for processing, the network monitoring messages amongst the plurality of collector processing sub-units based on a performance of one or more of the plurality of collector processing sub-units;
• generating, by the plurality of collector processing sub-units, network flow records from the received network monitoring messages, wherein the network flow records are generated in a first format and the network flow records are generated by augmenting the received network monitoring messages with additional information;
• transmitting, by the plurality of collector processing sub-units, the network flow records to a plurality of parser processing sub-units downstream from the collector processing sub-units;
• generating, by the plurality of parser processing sub-units, parsed network flow records by converting the received network flow records from the first format to a second format;
• combining, by the combiner operator, the parsed network flow records from the plurality of parser sub-units into a synchronous output data stream;
• and
• generating a profile store utilizing graphs, wherein the profiles from the profile store may be analyzed by either detection applications or machine learning, wherein the result of the analysis may be utilized for at least one of correlating applications, interacting with dashboards, or facilitating remedial network actions.

A computer program product comprising a non-transitory computer-readable medium having computer-executable instructions stored therein, the computer-executable instructions being executable by a processor to configure the processor to perform a method of processing network flow monitoring data, wherein the method comprises:

• receiving at least one incoming data stream of network monitoring messages;
• directing the network monitoring messages to a data processing segment comprising a load balancing operator, a data processing sequence that includes a plurality of collector processing sub-units and a plurality of parser sub-units, and a combiner operator, wherein the plurality of parser sub-units are directly connected to the plurality of collector processing sub-units, the plurality of parser sub-units are directly connected to the combiner operator, and the plurality of collector processing sub-units are directly connected to the load balancing operator such that data flows through the data processing segment sequentially from the load balancing operator to the plurality of collector processing sub-units to the plurality of parser sub-units to the combiner operator;
• distributing by the load balancing operator, for processing, the network monitoring messages amongst the plurality of collector processing sub-units based on a performance of one or more of the plurality of collector processing sub-units;
• generating, by the plurality of collector processing sub-units, network flow records from the received network monitoring messages, wherein the network flow records are generated in a first format and the network flow records are generated by augmenting the received network monitoring messages with additional information;
• transmitting, by the plurality of collector processing sub-units, the network flow records to a plurality of parser processing sub-units downstream from the collector processing sub-units;
• generating, by the plurality of parser processing sub-units, parsed network flow records by converting the received network flow records from the first format to a second format;
• combining, by the combiner operator, the parsed network flow records from the plurality of parser sub-units into a synchronous output data stream;
• and
• generating a profile store utilizing graphs, wherein the profiles from the profile store may be analyzed by either detection applications or machine learning, wherein the result of the analysis may be utilized for at least one of correlating applications, interacting with dashboards, or facilitating remedial network actions.