US 11,232,100 B2Granted

Resource allocation for multiple datasets

AssigneeSplunk Inc.

InventorsArindam Bhattacharjee, Sourav Pal, Christopher Pride

Life of this patent

Estimating the term…

Loading patent…

Abstract

Systems and methods are disclosed for processing queries against multiple dataset sources. One dataset source can include indexers that index and store data. The system can receive a query that identifies a set of data to be processed and a manner of processing the set of data. The set of data can include a first dataset that is accessible by one or more indexers and a second dataset that is accessible by one or more other dataset sources. A query coordinator can define a query processing scheme for obtaining and processing the set of data that includes a dynamic allocation of multiple layers of partitions. The partitions can operate on multiple worker nodes. The query can then be executed based on the query processing scheme.

Claims19

Dependent claims

A method comprising:

receiving, at a search service of a data intake and query system, a query identifying a set of data to be processed and a manner of processing the set of data, the set of data including a first dataset accessible by one or more indexers of the data intake and query system and a second dataset residing in an external data storage system communicatively coupled to the data intake and query system over a network;
defining, by the search service, a query processing scheme for obtaining and processing the set of data, the query processing scheme including:
first instructions to dynamically allocate a first set of processors from a plurality of processors to a first intake layer to obtain data from the one or more indexers and a second set of processors from the plurality of processors to a second intake layer to obtain data from the external data storage system,
second instructions for the one or more indexers to communicate first partial results corresponding to the first dataset to a first subset of the first set of processors,
third instructions directing a second subset of the second set of processors to communicate with the external data storage system to obtain the second dataset from the external data storage system;
fourth instructions directing the second subset of the second set of processors to process the second dataset based on the query processing scheme to obtain second partial results, and
fifth instructions directing the first subset of the first set of processors and the second subset of the second set of processors to combine the first partial results from the one or more indexers and the second partial results from the external data storage system to obtain results of the query and communicate the results of the query to the search service, wherein a quantity of processors for the second subset of the second set of processors is selected based on a determined capability of the external data storage system;
and executing the query based on the query processing scheme.

5 more elements

A computing system, comprising:

one or more processing devices configured to:
receive a query identifying a set of data to be processed and a manner of processing the set of data, the set of data including a first dataset accessible by one or more indexers of a data intake and query system and a second dataset residing in an external data storage system communicatively coupled to the data intake and query system over a network;
define a query processing scheme for obtaining and processing the set of data, the query processing scheme including:
first instructions to dynamically allocate a first set of processors from a plurality of processors to a first intake layer to obtain data from the one or more indexers and a second set of processors from the plurality of processors to a second intake layer to obtain data from the external data storage system,
second instructions for the one or more indexers to communicate first partial results corresponding to the first dataset to a first subset of the first set of processors,
third instructions directing a second subset of the second set of processors to communicate with the external data storage system to obtain the second dataset from the external data storage system;
fourth instructions directing the second subset of the second set of processors to process the second dataset based on the query processing scheme to obtain second partial results, and
fifth instructions directing the first subset of the first set of processors and the second subset of the second set of processors to combine the first partial results from the one or more indexers and the second partial results from the external data storage system to obtain results of the query and communicate the results of the query to the one or more processing devices, wherein a quantity of processors for the second subset of the second set of processors is selected based on a determined capability of the external data storage system;
and execute the query based on the query processing scheme.

6 more elements

Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:

receive a query identifying a set of data to be processed and a manner of processing the set of data, the set of data including a first dataset accessible by one or more indexers of a data intake and query system and a second dataset residing in an external data storage system communicatively coupled to the data intake and query system over a network;
define a query processing scheme for obtaining and processing the set of data, the query processing scheme including:
first instructions to dynamically allocate a first set of processors from a plurality of processors to a first intake layer to obtain data from the one or more indexers and a second set of processors from the plurality of processors to a second intake layer to obtain data from the external data storage system,
second instructions for the one or more indexers to communicate first partial results corresponding to the first dataset to a first subset of the first set of processors,
third instructions directing a second subset of the second set of processors to communicate with the external data storage system to obtain the second dataset from the external data storage system;
fourth instructions directing the second subset of the second set of processors to process the second dataset based on the query processing scheme to obtain second partial results, and
fifth instructions directing the first subset of the first set of processors and the second subset of the second set of processors to combine the first partial results from the one or more indexers and the second partial results from the external data storage system to obtain results of the query and communicate the results of the query to the computing system, wherein a quantity of processors for the second subset of the second set of processors is selected based on a determined capability of the external data storage system;
and execute the query based on the query processing scheme.

5 more elements

Dependencies are inferred from the claim text.

Facts

Application: 15/665,187Filed 31 Jul 2017 · continuation in part
Priority: 26 Sep 2016
Published: US 2018/0089258 A129 Mar 2018
Granted: 25 Jan 2022
Term ends: Not estimatedThe source records don’t support an estimate.

Classification4 codes

G06F 16/2425
Query formulation: iterative querying; Query formulation based on the results of a preceding query
G06F 16/2272
Indexing structures: management thereof
G06F 16/24535
Query rewriting; Transformation: of sub-queries or views
G06F
Computing and data processing

Family

Application15/276,717Parent of this continuation-in-part · filed 26 Sep 2016
US 2018/0089258 A1Pre-grant publication · published 29 Mar 2018
US 10,353,965 B2Parent of this continuation-in-part
US 2019/0163821 A1Published application
US 11,232,100 B2This patent

From the US filing history. Foreign family members aren’t listed.