Segmenting machine data into events based on source signatures

#This application claims benefit as a CON of U.S. Ser. No. 14/611,189, filed Jan. 31, 2015, which claims benefit as a CON of U.S. Ser. No. 14/170,228, filed Jan. 31, 2014, which claims benefit as a CON of U.S. Ser. No. 13/664,109, filed Oct. 30, 2012, now U.S. Pat. No. 8,694,450, issued Apr. 8, 2014, which claims benefit as a CON of U.S. Ser. No. 13/099,268, filed May 2, 2011, now U.S. Pat. No. 8,589,321, issued Nov. 19, 2013, which claims benefit as a CON of U.S. Ser. No. 11/459,632, filed Jul. 24, 2006, now U.S. Pat. No. 7,937,344, issued May 3, 2011, which claims benefit of U.S. PRO Ser. No. 60/702,496, filed Jul. 25, 2005, the entire contents of the aforementioned are hereby incorporated by reference as if fully set forth herein, under 35 U.S.C. § 120 The applicant(s) hereby rescind any disclaimer of claim scope in the parent application(s) or the prosecution history thereof and advise the USPTO that the claims in this application may be broader than any claim in the parent application(s).

#This invention relates generally to information organization and understanding, and more particularly to the organization and understanding of machine data.

#Information systems invariably generate vast amounts and wide varieties of machine data (e.g., activity logs, configuration files, messages, database records) whose value is widespread. Troubleshooting systems, detecting operational trends, catching security problems and measuring business performance, for example, typically require the organization and understanding of machine data. But the overwhelming volume, different and changing formats, and overall complexity of machine data create substantial difficulty for software developers, system administrators and business people who want to make sense of it and gain insight into information system behavior. The problem is compounded by the fact that information systems, and the machine data they generate, continue to grow in complexity and size.

#The invention has other advantages and features which will be more readily apparent from the following detailed description, when taken in conjunction with the accompanying drawings:

#FIG. 1 is a diagram an example information-processing environment suitable for use with an MDW.

#FIG. 2 is a flow diagram of one example of creation of an MDW according to the invention.

#In the example of FIG. 1, the information-processing environment includes hardware and software components such as computers, routers, databases, operating systems and applications in a distributed configuration for processing information. Each component may be producing MD 110, and there may be many MD sources and large quantities of MD across multiple technology and application domains. For example, a computer may be logging operating system events, a router may be auditing network traffic events, a database may be cataloging database reads and writes or schema changes, and an application may be sending the results of one application call to another across a message queue. In this embodiment, individual IT personnel—who may reside in different data centers, companies, or even geographies—typically manage specific technology and application domains. Aspects of the invention will be described with respect to the information-processing environments in FIG. 1, but the invention can also be used with other information-processing environments.

#FIG. 2 represents one approach 200 to building a MDW 290 from MD 110. This approach includes an organization process 235 and an understanding process 275. During the organization process 235, the MD 110 is organized into collections of discrete events 250, referred to herein as event data (ED). Events 250 represent units of system activity. Examples of events 250 include, for example, a web server servicing an HTTP “get” request from a web browser, an application server servicing an API call, or a database updating records in a table. Collections of events 250 can describe larger system activities, such as an attempt to update a customer record or submit an order. One of the challenges in organizing 235 MD 110 into events 250 is that MD generally has little formal structure and typically includes not much more than a time stamp common across different sources of MD and different types of events. MD 110 is also subject to changes in environment configurations. For example, changing the verbosity level in a web server configuration file can dramatically increase or decrease the amount of information included in an HTTP “get” event found in the web server's log file.

#During the understanding process 275, ED 250 is analyzed to create dynamic links between events and build the MDW 290. As an example, consider that a log from a web server may contain specific types of events 250 with specific event data, but a log from an application server or database may contain different events 250 and event data specific to its own domain. A system administrator may, for example, locate the web server event by looking for a session ID found in a web server log, locate the application server event by finding a process ID in the message queue, and locate a database table update event by searching for a transaction ID in the database audit trail. All three sources may contain events 250 that are part of a larger system activity, yet there is no obvious or explicit common structure or data shared among the MD 110 produced by each system. Common structure is manufactured across the three sources by analyzing the event data 250 so that connections between events can be identified. In one implementation, patterns of event behavior are recorded in real-time and identified, for example, as frequently occurring or infrequently occurring. Frequent patterns identify typical system processes and well-known links. Infrequent patterns identify deviations or anomalies and less well-known links. Contrast this with the world of the web, where hyperlinks are part of the formal, common structure of HTML—the language for building most web pages. Building links by hand for large volumes of ED 250 is not an option for complex information-processing environments.