# US 2020/0004794 A1 · Search functionality of a data intake and query system

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0004794 A1
- Document ID: US20200004794A1
- Status: Published application
- Kind: A1
- Application number: 16/570,545
- Filed: 2019-09-13
- Published: 2020-01-02
- Earliest claimed: 2016-09-26
- Assignees: Splunk Inc.
- Applicants: Splunk Inc.
- Inventors: Sourav Pal; Christopher Madden Pride; Arindam Bhattacharjee; Xiaowei Wang; James Alasdair Robert Hodge; Mustafa Ahamed
- Web page: https://patentgrep.com/patent/US20200004794A1

## CPC

- G06F 16/951: Indexing; Web crawling techniques
- G06F 16/211: {Schema design and management}
- G06F 16/212: {with details for data modelling support}
- G06F 16/2455: Query execution
- G06F 16/2471: {Distributed queries}
- G06F 16/248: Presentation of query results
- G06F 16/252: {between a Database Management System and a front-end application}
- G06F 16/258: {Data format conversion from or to a database}
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 16/9024: {Graphs; Linked lists (G06F16/9027 takes precedence)}
- G06F 16/90335: {Query processing}
- G06F 16/9038: Presentation of query results
- G06F 16/904: Browsing; Visualisation therefor (for navigating the web G06F16/954; browsing optimisation for the web G06F16/957)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Disclosed is a technique that can be performed in a distributed computer network. The technique can include a data index and query system that receives a search query, defines a search scheme for applying the search query on distributed data storage systems including an internal data storage system of the data index and query system and an external data storage system. The internal data storage system stores data as time-indexed events including respective segments of raw machine data. The data index and query system can transfer a portion of the search scheme to a search service, which can return search results obtained by application of the search scheme to the distributed data storage systems including the internal data storage system and the external data storage system. Lastly, the search results or data indicative of the search results can be output on a display device to the user.

## Claims

### Claim at position 1

-30. (canceled)

### Claim 31

A method comprising:
defining, by a data intake and query system, a search scheme for applying a search query on a plurality of distributed data storage systems including an internal data storage system of the data intake and query system and an external data storage system communicatively coupled to the data intake and query system over a computer network;
transferring, by the data intake and query system, a first portion of the search scheme to a search service;
receiving, by the data intake and query system from the search service, a first search result of the search query obtained by application of the first portion of the search scheme to at least the external data storage system of the plurality of distributed data storage systems;
producing, by the data intake and query system, a second search result of the search query by applying a second portion of the search scheme to the internal data storage system; and
causing, by the data intake and query system, the first and second search results or an output indicative of the first and second search results to be displayed by a display device.

### Claim 32 (depends on 31)

The method of claim 31, wherein the internal data storage system stores data as a plurality of time-indexed events, each of which includes a segment of raw machine data and an associated timestamp.

### Claim 33 (depends on 31)

The method of claim 31, further comprising:
transmitting, by the data intake and query system, the first search result to a worker node communicatively coupled to the external data storage system over the computer network.

### Claim 34 (depends on 31)

The method of claim 31, wherein the search service is implemented by a device that is communicatively coupled to the data intake and query system over the computer network.

### Claim 35 (depends on 31)

The method of claim 31, wherein the internal data storage system stores data in data stores of indexers that are accessible to the data intake and query system and the search service.

### Claim 36 (depends on 31)

The method of claim 31, further comprising:
receiving, by the data intake and query system, input indicative of the search query that is expressed in a pipelined language.

### Claim 37 (depends on 31)

The method of claim 31, further comprising:
receiving, by the data intake and query system, input indicative of the search query that is expressed in a structured language.

### Claim 38 (depends on 31)

The method of claim 31, wherein the output is rendered by the display device in a timeline visualization.

### Claim 39 (depends on 31)

The method of claim 31,
wherein the search query includes a parameter that causes the data intake and query system to define the first portion of the search scheme to be transferred to the search service, and
wherein the parameter is specified by a user responsible for submitting the search query.

### Claim 40 (depends on 31)

The method of claim 31, wherein defining the search scheme comprises:
determining that the search query requires searching the plurality of distributed data storage systems; and
defining a plurality of phases based on the search query, the plurality of phases including:
a first phase for searching the plurality of distributed data storage systems, and
a second phase for processing search results obtained in the first phase, wherein the first portion of the search scheme includes at least one of the plurality of phases.

### Claim 41 (depends on 31)

The method of claim 31, wherein defining the search scheme comprises:
determining that the search query requires searching the plurality of distributed data storage systems; and
defining a plurality of phases based on the search query, the plurality of phases including:
a first phase for searching the plurality of distributed data storage systems, and
a second phase for processing search results obtained in the first phase,
wherein the first portion of the search scheme includes at least one of the plurality of phases, and
wherein the first phase is performed by a plurality of indexers executing in parallel.

### Claim 42 (depends on 31)

The method of claim 31, wherein defining the search scheme comprises:
determining that the search query requires searching the plurality of distributed data storage systems; and
defining a plurality of ordered phases based on the search query, the plurality of ordered phases including:
a first phase for searching the plurality of distributed data storage systems, and
a second phase for processing search results obtained in the first phase,
wherein the first portion of the search scheme includes at least one of the plurality of ordered phases.

### Claim 43 (depends on 31)

The method of claim 31, further comprising:
executing, by the data intake and query system, a first search operation on the internal data storage system in accordance with the search scheme; and
sending, by the data intake and query system to the search service, a message that causes the search service to initiate a second search operation for searching the external storage system in accordance with the search scheme.

### Claim 44 (depends on 31)

The method of claim 31, wherein the first search result is based on structured data stored in the external data storage system.

### Claim 45 (depends on 31)

The method of claim 31, wherein the second search result is based on unstructured data stored in the internal data storage system.

### Claim 46 (depends on 31)

The method of claim 31, wherein the first search result, the second search result, or both are in a format specified by the search scheme.

### Claim 47 (depends on 31)

The method of claim 31,
wherein the first search result is based on structured data stored in the external data storage system,
wherein the second search result is based on unstructured data stored in the internal data storage system, and
wherein the method further comprises:
transforming, by the data intake and query system, the first and second search results into a common format to facilitate aggregation of partial search results obtained in response to the search query.

### Claim 48 (depends on 31)

The method of claim 31,
wherein the first search result is based on structured data stored in the external data storage system,
wherein the second search result is based on unstructured data stored in the internal data storage system, and
wherein the method further comprises:
transforming, by the data intake and query system, the first and second search results into a common format to facilitate aggregation of partial search results obtained in response to the search query, wherein the first and second search results are transformed into data events having structures that are compatible with the data intake and query system.

### Claim 49 (depends on 31)

The method of claim 31, wherein the search query includes a parameter that causes the data intake and query system to define the first portion of the search scheme.

### Claim 50 (depends on 31)

The method of claim 31, wherein the search query includes a parameter that cause the data intake and query system to define the first portion of the search scheme, and wherein the parameter is specified by a user responsible for submitting the search query.

### Claim 51 (depends on 31)

The method of claim 31, wherein the first portion of the search scheme transferred to the search service includes a plurality of phases for obtaining data from the plurality of distributed data storage systems.

### Claim 52 (depends on 31)

The method of claim 31, wherein the first portion of the search scheme transferred to the search service includes a plurality of ordered phases for obtaining data from the plurality of distributed data storage systems in accordance with an order of the plurality of ordered phases.

### Claim 53 (depends on 31)

The method of claim 31, wherein the first search result of the search query is obtained by application of the first portion of the search scheme to the external data storage system and the internal data storage system.

### Claim 54 (depends on 31)

The method of claim 31,
wherein the first search result of the search query is obtained by application of the first portion of the search scheme to the external data storage system and the internal data storage system, and
wherein the first search result obtained by the data intake and query system from the search service corresponds to a combination of partial search results of the external data storage system and partial search results of the internal data storage system.

### Claim 55 (depends on 31)

The method of claim 31,
wherein the first search result of the search query is obtained by application of the first portion of the search scheme to the external data storage system and the internal data storage system,
wherein the first search result obtained by the data intake and query system from the search service corresponds to a combination of partial search results of the external data storage system and partial search results of the internal data storage system, and
wherein the first search result is formed by combining the partial search results of the external data storage system and the partial search results of the internal data storage system as an arrangement of time-ordered events.

### Claim 56 (depends on 31)

The method of claim 31,
wherein the first search result of the search query is obtained by application of the first portion of the search scheme to the external data storage system and the internal data storage system, and
wherein the first search result obtained by the data intake and query system from the search service is derived from a combination of partial search results of the external data storage system and partial search results of the internal data storage system.

### Claim 57 (depends on 31)

The method of claim 31,
wherein the search scheme includes a plurality of phases, and
wherein upon receiving the first portion of the search scheme, the search service defines an executable search process based on the first portion of the search scheme.

### Claim 58 (depends on 31)

The method of claim 31,
wherein the search scheme includes a plurality of phases,
wherein upon receiving the first portion of the search scheme, the search service defines an executable search process based on the first portion of the search scheme, and
wherein the executable search process is defined as a directed acyclic graph that includes at least one of the plurality of phases.

### Claim 59

A data intake and query system comprising:
a processor; and
a memory having instructions executable by the processor to cause the data intake and query system to:
define a search scheme for applying a search query on a plurality of distributed data storage systems including an internal data storage system of the data intake and query system and an external data storage system communicatively coupled to the data intake and query system over a computer network;
transfer a first portion of the search scheme to a search service;
receive, from the search service, a first search result of the search query obtained by application of the first portion of the search scheme to at least the external data storage system of the plurality of distributed data storage systems;
produce a second search result of the search query by applying a second portion of the search scheme to the internal data storage system; and
cause the first and second search results or data indicative of the first and second search results to be displayed by a display device.

### Claim 60

Non-transitory computer-readable media including computer-executable instructions that, when executed by a computing system, cause the computing system to perform operations comprising:
defining a search scheme for applying a search query on a plurality of distributed data storage systems including an internal data storage system of the data intake and query system and an external data storage system communicatively coupled to the data intake and query system over a computer network;
transferring a first portion of the search scheme to a search service;
receiving, from the search service, a first search result of the search query obtained by application of the first portion of the search scheme to at least the external data storage system of the plurality of distributed data storage systems;
producing a second search result of the search query by applying a second portion of the search scheme to the internal data storage system; and
causing the first and second search results or data indicative of the first and second search results to be displayed by a display device.
