Leveraging point inferences on HTTP transactions for HTTPS malware detection

An apparatus, comprising:

• one or more network interfaces to communicate with a network;
• a processor coupled to the network interfaces and configured to execute one or more processes;
• and
• a memory configured to store a process executable by the processor, the process when executed configured to:
• receive captured traffic data regarding a Transport Layer Security (TLS) connection between a client and a server;
• apply a first machine learning-based classifier to TLS records from the traffic data to label each of the TLS records as either a TLS record that includes Hypertext Transfer Protocol (HTTP) header information comprising one or more HTTP header fields or a TLS record that does not include the HTTP header information;
• estimate one or more HTTP transaction labels for the connection by applying a second machine learning-based classifier to the identified set of TLS records that include the HTTP header information;
• augment the captured traffic data with the one or more HTTP transaction labels;
• and
• cause performance of a network security function based on the augmented traffic data,
• wherein the TLS records remain encrypted during classification by the first and second machine learning-based classifiers.

A tangible, non-transitory, computer-readable medium storing program instructions that cause a traffic analysis service to execute a process comprising:

• receiving, at the traffic analysis service, captured traffic data regarding a Transport Layer Security (TLS) connection between a client and a server;
• applying, by the traffic analysis service, a first machine learning-based classifier to TLS records from the traffic data to label each of the TLS records as either a TLS record that includes Hypertext Transfer Protocol (HTTP) header information comprising one or more HTTP header fields or a TLS record that does not include the HTTP header information;
• estimating, by the traffic analysis service, one or more HTTP transaction labels for the connection by applying a second machine learning-based classifier to the identified set of TLS records that include the HTTP header information;
• augmenting, by the traffic analysis service, the captured traffic data with the one or more HTTP transaction labels;
• and
• causing, by the traffic analysis service, performance of a network security function based on the augmented traffic data,
• wherein the TLS records remain encrypted during classification by the first and second machine learning-based classifiers.

A method for classifying an encrypted flow, comprising:

• receiving, at a network device, a plurality of packets associated with a first encrypted flow traversing a network;
• collecting telemetry data from the plurality of packets associated with the first encrypted flow, wherein at least a portion of the telemetry data is collected from packets encrypted according to a cryptographic protocol, wherein the telemetry data includes cryptographic protocol data, and wherein the telemetry data further includes at least one of a source IP address, a destination IP address, a destination port, a start time of the first encrypted flow, a stop time of the first encrypted flow, a protocol associated with the first encrypted flow, a number of packets of the first encrypted flow, a number of bytes of the first encrypted flow, a sequence of TLS record lengths and types, a sequence of TCP packet lengths and flags, a number of bytes extracted from the first encrypted flow, a sequence of packet lengths and times from the first encrypted flow, a series of application lengths and times from the first encrypted flow, or byte distribution data;
• and
• sending the telemetry data to a traffic analysis process,
• wherein the traffic analysis process comprises:
• using the telemetry data in conjunction with a first machine learning classifier to make a first assessment whether the first encrypted flow is part of a set of traffic flows caused by malware in the network,
• using the first assessment as an input to a second machine learning process that generates a classification of the first encrypted flow, and
• using the classification of the first encrypted flow from the second machine learning process to modify a processing of the first encrypted flow.

A method for creating and operating a multi-step classifier for an encrypted flow, the method comprising:

• collecting telemetry data from a first plurality of encrypted flows using a plurality of traffic capture services located on network paths between clients and servers, the first plurality of encrypted flows including both malicious and benign flows, wherein at least a portion of the telemetry data is collected from packets encrypted according to a cryptographic protocol, wherein the telemetry data includes cryptographic protocol data, and wherein the telemetry data further includes at least one of a source IP address, a destination IP address, a destination port, a start time of any of the first plurality of encrypted flows, a stop time of any of the first plurality of encrypted flows, a protocol associated with any of the first plurality of encrypted flows, a number of packets of any of the first plurality of encrypted flows, a number of bytes of any of the first plurality of encrypted flows, a sequence of TLS record lengths and types, a sequence of TCP packet lengths and flags, a number of bytes extracted from any of the first plurality of encrypted flows, a sequence of packet lengths and times from any of the first plurality of encrypted flows, a series of application lengths and times from any of the first plurality of encrypted flows, or byte distribution data;
• using the telemetry data to create a first trained machine learning classifier to generate a classification of features present in the first plurality of encrypted flows;
• using the telemetry data to create a second trained machine learning classifier to classify flows as malicious or benign, based at least in part on the classification of features from the first trained machine learning classifier;
• receiving second telemetry data from an unclassified encrypted flow;
• using the first trained machine learning classifier to make an inference regarding the unclassified encrypted flow;
• and
• using the second trained machine learning classifier to classify the unclassified encrypted flow as malicious or benign using the second telemetry data and the inference made by the first trained machine learning classifier.