# US 2019/0171629 A1 · Segmenting machine data using token-based signatures > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2019/0171629 A1 - Document ID: US20190171629A1 - Status: Published application - Kind: A1 - Application number: 16/264,610 - Filed: 2019-01-31 - Published: 2019-06-06 - Earliest claimed: 2006-10-05 - Assignees: Splunk Inc. - Applicants: Splunk Inc. - Inventors: Michael Joseph BAUM; R. David CARASSO; Robin Kumar DAS; Rory GREENE; Bradley HALL; Nicholas Christian MEALY; Brian Philip MURPHY; Stephen Phillip SORKIN; Andre David STECHERT; Erik M. SWAN - Web page: https://patentgrep.com/patent/US20190171629A1 ## CPC - G06F 16/2272: {Management thereof} - G06F 16/2228: {Indexing structures} - G06F 16/2291: {User-Defined Types; Storage management thereof} - G06F 16/2322: {using timestamps} - G06F 16/24568: {Data stream processing; Continuous queries} - G06F 16/24575: {using context} - G06F 16/24578: {using ranking} - G06F 16/2477: {Temporal data queries} - G06F 16/248: Presentation of query results - G06F 16/951: Indexing; Web crawling techniques - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods and apparatus consistent with the invention provide the ability to organize, index, search, and present time series data based on searches. Time series data are sequences of time stamped records occurring in one or more usually continuous streams, representing some type of activity. In one embodiment, time series data is organized into discrete events with normalized time stamps and the events are indexed by time and keyword. A search is received and relevant event information is retrieved based in whole or in part on the time indexing mechanism, keyword indexing mechanism, or statistical indices calculated at the time of the search. ## Claims ### Claim 1 A method, comprising: analyzing one or more tokens that appear in a portion of a set of machine data; mapping the one or more tokens to a frequency of occurrence in the portion of the set of machine data; creating a sample signature using the frequency of occurrence of the one or more tokens; determining a source of the set of machine data based on a comparison of the sample signature with signatures in a set of signatures from known sources; segmenting the machine data into a plurality of events using a set of rules corresponding to the determined source thereby allowing application of time-based search phrases across the segmented machine data in the plurality of events; wherein the method is performed by one or more computing devices. ### Claim 2 (depends on 1) The method of claim 1, further comprising: creating a signature for a known source by creating a source signature from machine data collected from the known source; storing the signature in the set of signatures. ### Claim 3 (depends on 1) The method of claim 1, wherein each event in the plurality of events includes a portion of the set of machine data. ### Claim 4 (depends on 1) The method of claim 1, wherein each event in the plurality of events includes a portion of the set of machine data, and wherein each event is field-searchable. ### Claim 5 (depends on 1) The method of claim 1, further comprising: creating a time stamp for each event in the plurality of events by extracting time stamp information from machine data included in each event; ### Claim 6 (depends on 1) The method of claim 1, wherein the determining a source of the set of machine data further comprises: matching the sample signature to signatures in the set of signatures from known sources using a nearest neighbor search. ### Claim 7 (depends on 1) The method of claim 1, further comprising: based on a determination that the source of the machine data is not known: defining a new source for the set of machine data; creating a new signature for the new source using the sample signature for the machine data. ### Claim 8 (depends on 1) The method of claim 1, further comprising: based on a determination that the source of the machine data is not known: setting the determined source for the set of machine data to a default source. ### Claim 9 (depends on 1) The method of claim 1, further comprising: determining that the machine data is binary machine data; converting the binary machine data into textual machine data using a process defined for the determined source. ### Claim 10 (depends on 1) The method of claim 1, further comprising: determining that the machine data is binary machine data by comparing a number of machine data lines in the machine data that appear to be binary and a number of machine data lines in the machine data that appear to be textual; converting the binary machine data into textual machine data using a process defined for the determined source. ### Claim 11 One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause performance of: analyzing one or more tokens that appear in a portion of a set of machine data; mapping the one or more tokens to a frequency of occurrence in the portion of the set of machine data; creating a sample signature using the frequency of occurrence of the one or more tokens; determining a source of the set of machine data based on a comparison of the sample signature with signatures in a set of signatures from known sources; segmenting the machine data into a plurality of events using a set of rules corresponding to the determined source thereby allowing application of time-based search phrases across the segmented machine data in the plurality of events. ### Claim 12 (depends on 11) The one or more non-transitory computer-readable storage media of claim 11, further comprising: creating a signature for a known source by creating a source signature from machine data collected from the known source; storing the signature in the set of signatures. ### Claim 13 (depends on 11) The one or more non-transitory computer-readable storage media of claim 11, wherein each event in the plurality of events includes a portion of the set of machine data. ### Claim 14 (depends on 11) The one or more non-transitory computer-readable storage media of claim 11, wherein each event in the plurality of events includes a portion of the set of machine data, and wherein each event is field-searchable. ### Claim 15 (depends on 11) The one or more non-transitory computer-readable storage media of claim 11, further comprising: creating a time stamp for each event in the plurality of events by extracting time stamp information from machine data included in each event; ### Claim 16 (depends on 11) The one or more non-transitory computer-readable storage media of claim 11, wherein the determining a source of the set of machine data further comprises: matching the sample signature to signatures in the set of signatures from known sources using a nearest neighbor search. ### Claim 17 (depends on 11) The one or more non-transitory computer-readable storage media of claim 11, further comprising: based on a determination that the source of the machine data is not known: defining a new source for the set of machine data; creating a new signature for the new source using the sample signature for the machine data. ### Claim 18 (depends on 11) The one or more non-transitory computer-readable storage media of claim 11, further comprising: based on a determination that the source of the machine data is not known: setting the determined source for the set of machine data to a default source. ### Claim 19 An apparatus, comprising: a signature creation device, implemented at least partially in hardware, that analyzes one or more tokens that appear in a portion of a set of machine data; wherein the signature creation device maps the one or more tokens to a frequency of occurrence in the portion of the set of machine data; wherein the signature creation device creates a sample signature using the frequency of occurrence of the one or more tokens; a signature comparison device, implemented at least partially in hardware, that determines a source of the set of machine data based on a comparison of the sample signature with signatures in a set of signatures from known sources; an event creation device, implemented at least partially in hardware, that segments the machine data into a plurality of events using a set of rules corresponding to the determined source thereby allowing application of time-based search phrases across the segmented machine data in the plurality of events. ### Claim 20 (depends on 19) The apparatus of claim 19, further comprising: wherein the signature creation device creates a signature for a known source by creating a source signature from machine data collected from the known source; a signature storage device, implemented at least partially in hardware, that stores the signature in the set of signatures.