Log collection, structuring and processing

A method for use in monitoring one or more platforms of a data system, comprising the steps of:

• identifying, using a processing platform, a log message to be archived, wherein log messages in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols;
• first operating said processing platform to associate metadata with each of multiple parsed portions of the said identified log message to be archived, wherein said metadata includes information to assist in restoring said message;
• and
• second operating said processing platform to archive said log message together with said metadata in a data structure for enabling restoration of said log message using said metadata.

A method for use in processing logs in a data system, comprising the steps of:

• providing a computer-based tool for use in accessing archived logs based on metadata describing each of multiple parsed portions of the logs, wherein log messages in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols, wherein the multiple parsed portions comprise data fields corresponding to one or more of a log host, a log message source, an IP address, a program and a login;
• first operating said computer-based tool to receive restoration information associated with one or more fields of said metadata;
• and
• second operating said computer-based tool to restore one or more logs based on said received restoration information.

A method for use in monitoring one or more platforms of a data system, comprising the steps of:

• establishing an agent protocol defining communications between an agent for acquiring log messages and a processor for processing log messages, wherein log messages in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols;
• providing an agent for monitoring log messages associated with a monitored platform;
• and
• operating said agent to identify a log message of interest, process said log message to parse said message and associate metadata with parsed portions of said message, and provide an output in accordance with said agent protocol, wherein said output includes substantially an entirety of a content of said message together with said metadata.

A method for use in monitoring one or more platforms of a data system, comprising the steps of:

• establishing an agent protocol defining communications between an agent for acquiring log messages and a processor for processing log messages, wherein log messages in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols;
• receiving, at said processor, a pre-processed log message in accordance with said protocol, where said pre-processed log message includes parsed portions of a raw log message, collectively including substantially an entirety of a content of said raw message, together with metadata defining said parsed portions;
• and
• operating said processor to use said metadata to further process said preprocessed log message.

A method for use in monitoring one or more platforms in a data system, comprising the steps of:

• first operating a processing platform to access a Windows event log from a monitored platform;
• wherein event logs in different formats are generated by said one or more platforms and are collected using standard system logging and messaging protocols;
• second operating said processing platform to identify a timestamp associated with the Windows event log;
• and
• third operating said processing platform to normalize the timestamp such that the time stamp is substantially independent of a processing environment of the monitored platform;
• wherein said step of third operating comprises accounting for one of a local time zone, a local clock offset, and a local platform time system.