Automatic parsing of binary-based application protocols using network traffic

A system for analyzing a protocol of a network, comprising:

• an acquisition module configured to:
• obtain a plurality of conversations from the network, wherein each of the plurality of conversations comprises a sequence of messages exchanged between a server and a client of the network using the protocol, wherein each message of the sequence of messages comprises a plurality of fields, wherein a field of the plurality field is located, within a corresponding message, at an offset and having a length that are defined by the protocol;
• a protocol field extractor executing on a processor of a computer system and configured to:
• extract content of a candidate field from a message of the sequence of messages in each of the plurality of conversations, wherein the candidate field is located, within the message, at a candidate offset and having a candidate length;
• calculate a randomness measure of the content of the candidate field, wherein the randomness measure represents a level of randomness of the content across all of the plurality of conversations;
• calculate a correlation measure of the content of the candidate field, wherein the correlation measure represents a level of correlation, across all of the plurality of conversations, between the content and an attribute of a corresponding conversation where the message containing the candidate field is located;
• and
• select, based on the randomness measure and the correlation measure, and using a pre-determined field selection criterion, the candidate offset from a plurality of candidate offsets as the offset defined by the protocol;
• and
• a repository configured to store the plurality of conversations, the randomness measure, and the correlation measure.

The system of claim 12,

• wherein the field is the message-length field for representing a length of the corresponding message wherein the field is located,
• wherein the attribute comprises a length indication of a message where the candidate field is located,
• wherein selecting the candidate offset as the offset defined by the protocol comprises comparing the randomness measure and the correlation measure to a pre-determined randomness threshold and a pre-determined correlation threshold, respectively, and
• wherein the candidate offset is selected as the offset defined by the protocol in response to the randomness measure exceeding the pre-determined randomness threshold and the correlation measure exceeding the pre-determined correlation threshold.

A non-transitory computer readable medium embodying instructions for analyzing a protocol of a network, the instructions when executed by a processor comprising functionality for:

• obtaining a plurality of conversations from the network, wherein each of the plurality of conversations comprises a sequence of messages exchanged between a server and a client of the network using the protocol, wherein each message of the sequence of messages comprises a plurality of fields, wherein a field of the plurality field is located, within a corresponding message, at an offset and having a length that are defined by the protocol;
• extracting content of a candidate field from a message of the sequence of messages in each of the plurality of conversations, wherein the candidate field is located, within the message, at a candidate offset and having a candidate length;
• calculating a randomness measure of the content of the candidate field, wherein the randomness measure represents a level of randomness of the content across all of the plurality of conversations;
• calculating a correlation measure of the content of the candidate field, wherein the correlation measure represents a level of correlation, across all of the plurality of conversations, between the content and an attribute of a corresponding conversation where the message containing the candidate field is located;
• and
• selecting based on the randomness measure and the correlation measure, and using a pre-determined field selection criterion, the candidate offset from a plurality of candidate offsets as the offset defined by the protocol.