Machine learning based botnet detection using real-time extracted traffic features

A system for identifying a botnet in a network, comprising:

• a processor;
• and
• memory storing instructions, when executed by the processor, comprising functionalities for:
• obtaining historical network data in the network, the historical network data comprising a first plurality of data units;
• obtaining a ground truth data set associated with the historical network data, the ground truth data set comprising a plurality of labels with each label assigned to a corresponding data unit of the first plurality of data units, said each label comprising one of a first label categorizing said corresponding data unit as associated with the botnet and a second label categorizing said corresponding data unit as being not associated with the botnet;
• analyzing, using a machine learning algorithm, the historical network data and the ground truth data set to generate a model comprising a decision tree and a plurality of functions each associated with one of a plurality of leaf nodes of the decision tree, wherein the decision tree comprises the plurality of leaf nodes, a first non-leaf node, and a second non-leaf node downstream from the first non-leaf node, wherein the historical network data is applied as input to the decision tree and is split while traversing the decision tree, wherein the first non-leaf node splits a first portion of the historical network data to traverse two paths of the decision tree, wherein the first portion of the historical network data is split based on a packet inter-arrival time statistics of the first plurality of data units, wherein the second non-leaf node splits a second portion of the historical network data traversing one of the two paths of the decision tree to the second non-leaf node, wherein the second portion of the historical network data is split based on at least one selected from a group consisting of a layer-4 payload size statistics and a number of bytes per packet statistics of the first plurality of data units, wherein each of the plurality of functions outputs a first statistical prediction of the plurality of labels assigned to a third portion of the historical network data traversing the decision tree to a corresponding leaf node of the plurality of leaf nodes, wherein the machine learning algorithm adjusts the model to match the first statistical prediction to the ground truth data set;
• obtaining real-time network data in the network, the real-time network data comprising a second plurality of data units;
• applying the second plurality of data units as the input to the decision tree to output a second statistical prediction of the plurality of labels assigned to a second data unit of the second plurality of data units;
• and
• categorizing the second data unit as associated with the botnet based on the second statistical prediction.

A non-transitory computer readable medium storing instructions for identifying a botnet in a network, the instructions, when executed by a processor of a computer, comprising functionality for:

• obtaining historical network data in the network, the historical network data comprising a first plurality of data units;
• obtaining a ground truth data set associated with the historical network data, the ground truth data set comprising a plurality of labels with each label assigned to a corresponding data unit of the first plurality of data units, said each label comprising one of a first label categorizing said corresponding data unit as associated with the botnet and a second label categorizing said corresponding data unit as not associated with the botnet;
• analyzing, using a machine learning algorithm, the historical network data and the ground truth data set to generate a model comprising a decision tree and a plurality of functions each associated with one of a plurality of leaf nodes of the decision tree, wherein the decision tree comprises the plurality of leaf nodes, a first non-leaf node, and a second non-leaf node downstream from the first non-leaf node, wherein the historical network data is applied as input to the decision tree and is split while traversing the decision tree, wherein the first non-leaf node splits a first portion of the historical network data to traverse two paths of the decision tree, wherein the first portion of the historical network data is split based on a packet inter-arrival time statistics of the first plurality of data units, wherein the second non-leaf node splits a second portion of the historical network data traversing one of the two paths of the decision tree to the second non-leaf node, wherein the second portion of the historical network data is split based on at least one selected from a group consisting of a layer-4 payload size statistics and a number of bytes per packet statistics of the first plurality of data units, wherein each of the plurality of functions outputs a first statistical prediction of the plurality of labels assigned to a third portion of the historical network data traversing the decision tree to a corresponding leaf node of the plurality of leaf nodes, wherein the machine learning algorithm adjusts the model to match the first statistical prediction to the ground truth data set;
• obtaining real-time network data in the network, the real-time network data comprising a second plurality of data units;
• applying the second plurality of data units as the input to the decision tree to output a second statistical prediction of the plurality of labels assigned to a second data unit of the second plurality of data units;
• and
• categorizing the second data unit as associated with the botnet based on the second statistical prediction.