Processing of computer log messages for visualization and retrieval

A system for processing computer log messages for log visualization and log retrieval to improve log processing and management systems, comprising:

• a memory device for storing program code;
• at least one processor device operatively coupled to the memory device and configured to execute program code stored on the memory device to:
• collect a plurality of log messages from one or more computer system components to obtain collected log messages;
• perform a log tokenization process on the collected log messages to generate tokens;
• transform the tokens into collected log vectors associated with a metric space by, for each collected log message, averaging vector representations in the metric space to obtain a d-dimensional vector representation for the collected log message;
• perform a locality preserving dimensionality reduction technique on the metric space to project the metric space into a lower dimensional sub-space;
• store similarity distances between respective pairs of the log vectors;
• receive a query associated with a query log message for reducing operational inefficiencies of the one or more computer system components;
• determine which cluster of a plurality of clusters a query log vector corresponding to the query log message belongs to based on a cluster identification (ID) of the query log vector to obtain an identified cluster, wherein the collected log vectors are assigned cluster IDs corresponding to their respective clusters, and wherein each of the plurality of clusters includes at least one collected log vector with at least one of the plurality of clusters including at least two collected log vectors;
• and
• retrieve one or more similar log messages corresponding to the query log message for reducing the operational inefficiencies of the one or more computer system components based on a ranking of respective similarity distances between the query log vector of the identified cluster and the at least one collected log vector of the identified cluster.

A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for processing computer log messages for log visualization and log retrieval to improve log processing and management systems, the method performed by the computer comprising:

• collecting a plurality of log messages from one or more computer system components to obtain collected log messages;
• performing a log tokenization process on the collected log messages to generate tokens;
• transforming the tokens into collected log vectors associated with a metric space including, for each collected log message, averaging vector representations in the metric space to obtain a d-dimensional vector representation for the collected log message;
• performing a locality preserving dimensionality reduction technique on the metric space to project the metric space into a lower dimensional sub-space;
• storing similarity distances between respective pairs of the log vectors;
• receiving a query associated with a query log message for reducing operational inefficiencies of the one or more computer system components;
• determining which cluster of a plurality of clusters a query log vector corresponding to the query log message belongs to based on a cluster identification (ID) of the query log vector to obtain an identified cluster, wherein the collected log vectors are assigned cluster IDs corresponding to their respective clusters, and wherein each of the plurality of clusters includes at least one collected log vector with at least one of the plurality of clusters including at least two collected log vectors;
• and
• retrieving one or more similar log messages corresponding to the query log message for reducing the operational inefficiencies of the one or more computer system components based on a ranking of respective similarity distances between the query log vector of the identified cluster and the at least one collected log vector of the identified cluster.