Compression of log data using field types

A system, comprising:

• a memory;
• and
• at least one processing device, coupled to the memory, operative to implement the following steps:
• obtaining at least one log message of a plurality of log messages, wherein the at least one log message comprises a message template and one or more message variable values, wherein the message template is repeated across the plurality of log messages and comprises at least one field that is replaced with at least one of the one or more message variable values;
• incrementing a counter, associated with the message template, indicating a frequency of occurrence of the message template, wherein the counter is one of a plurality of counters each associated with a respective one of a plurality of different message templates;
• obtaining a compression index that maps each of the plurality of different message templates to a corresponding message signature of a plurality of message signatures, wherein the mapping assigns a given message signature to a particular message template based on a length of the given message signature and a frequency of occurrence of the particular message template;
• identifying a predefined field type of each of the one or more message variable values;
• selecting, using the at least one processing device, a compression technique to apply to each of the one or more message variable values based on the corresponding identified predefined field type to obtain one or more compressed message variable values, wherein the selected compression technique compresses the one or more message variable values using one or more of:
• (i) a difference between a predicted next value of the one or more message variable values and an actual value of the one or more message variable values and (ii) a difference between a current value of the one or more message variable values and a base value of the one or more message variable values;
• and
• writing, using the at least one processing device, the one or more compressed message variable values and a message signature corresponding to the message template of the at least one log message to a log file.

A computer program product, comprising a non-transitory machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:

• obtaining at least one log message of a plurality of log messages, wherein the at least one log message comprises a message template and one or more message variable values, wherein the message template is repeated across the plurality of log messages and comprises at least one field that is replaced with at least one of the one or more message variable values;
• incrementing a counter, associated with the message template, indicating a frequency of occurrence of the message template, wherein the counter is one of a plurality of counters each associated with a respective one of a plurality of different message templates;
• obtaining a compression index that maps each of the plurality of different message templates to a corresponding message signature of a plurality of message signatures, wherein the mapping assigns a given message signature to a particular message template based on a length of the given message signature and a frequency of occurrence of the particular message template;
• identifying a predefined field type of each of the one or more message variable values;
• selecting, using the at least one processing device, a compression technique to apply to each of the one or more message variable values based on the corresponding identified predefined field type to obtain one or more compressed message variable values, wherein the selected compression technique compresses the one or more message variable values using one or more of:
• (i) a difference between a predicted next value of the one or more message variable values and an actual value of the one or more message variable values and (ii) a difference between a current value of the one or more message variable values and a base value of the one or more message variable values;
• and
• writing, using the at least one processing device, the one or more compressed message variable values and a message signature corresponding to the message template of the at least one log message to a log file.