Detecting malicious behavior from handshake protocols using machine learning

A malware detection system comprising:

• one or more processors;
• and
• memory having instructions stored thereon that, when executed by the one or more processors, cause the system to:
• execute, within a secure operating environment, a plurality of known malicious software applications and a plurality of known non-malicious software applications, whereby the plurality of known malicious software applications and the plurality of known non-malicious software applications initiate respective connections with one or more target servers from the secure operating environment upon execution;
• generate a dataset of known handshake parameters corresponding to the plurality of known malicious software applications and the plurality of known non-malicious software applications, wherein for each respective connection between the plurality of known malicious software applications and one or more target servers or the plurality of known non-malicious software applications and the one or more target servers, a set of handshake parameters is collected, the set of handshake parameters including:
• a first subset of parameters transmitted from a known malicious software application or a known non-malicious software application to the one or more target servers in a client channel setup message;
• and
• a second subset of parameters transmitted from the target server to the known malicious software application or the known non-malicious software application in a server channel setup message;
• wherein generating comprises evaluating non-numerical data from the first subset of parameters and the second subset of parameters;
• train a machine learning model using the dataset of known handshake parameters, wherein the machine learning model is configured to predict a maliciousness of a connection between two or more computing devices based on handshake parameters between the two or more computing devices;
• and
• distribute the machine learning model to one or more client devices.

A non-transitory computer readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• executing, within a secure testing environment, a plurality of known malicious software applications and a plurality of known non-malicious software applications, whereby the plurality of known malicious software applications and the plurality of known non-malicious software applications initiate respective connections with one or more target servers from the secure testing environment upon execution;
• generating a dataset of known handshake parameters corresponding to the plurality of known malicious software applications and the plurality of known non-malicious software applications, wherein for each respective connection between the plurality of known malicious software applications and one or more target servers or the plurality of known non-malicious software applications and the one or more target servers, a set of handshake parameters is collected, the set of handshake parameters including:
• a first subset of parameters transmitted from a known malicious software application or a known non-malicious software application to the one or more target servers in a client channel setup message;
• and
• a second subset of parameters transmitted from the target server to the known malicious software application or the known non-malicious software application in a server channel setup message;
• wherein generating comprises evaluating non-numerical data from the first subset of parameters and the second subset of parameters;
• training a machine learning model using the dataset of known handshake parameters, wherein the machine learning model is configured to predict a maliciousness of a connection between two or more computing devices based on handshake parameters between the two or more computing devices;
• and
• distributing the machine learning model to one or more client devices.