Query execution at a remote heterogeneous data store of a data fabric service
AssigneeSplunk Inc.
InventorsSourav Pal, Arindam Bhattacharjee, Timothy Tully
Life of this patent
Estimating the term…
Abstract
Systems and methods are described for executing a query of raw machine data that is stored at a remote data store that may store heterogeneous data. The system can determine the directories or file types that may store event data and may instruct one or more worker nodes to access files that may store events based on the determined directories of file types. Further, the system may exclude files at the remote data store that may not be identified as potentially storing events enabling a query that implicates a heterogeneous data store to be efficiently executed.
Claims30
A method, comprising:
- receiving a query that references a remote data store;
- accessing a configuration file to identify directories and a plurality of file types that are to be accessed in the remote data store, wherein the remote data store stores heterogeneous data in files having different file types;
- allocating a plurality of processors to communicate with the remote data store;
- generating instructions for worker nodes, wherein the generated instructions include instructions to:
- scan, using the plurality of processors, the identified directories;
- open, using a processor of the plurality of processors, a file in the remote data store, the file having a file type of the plurality of file types identified in the configuration file;
- apply, using the processor, filter criteria to content of the file;
- and
- include, using the processor, data from the file that satisfies the filter criteria in a partition, wherein the plurality of processors concurrently process a plurality of partitions including the partition, wherein each partition comprises data from one or more files of the remote data store;
- and
- executing a query based on the generated instructions.
A computing system of a data intake and query system, the computing system comprising:
- memory;
- and
- one or more processing devices coupled to the memory and configured to:
- receive a query that references a remote data store;
- access a configuration file to identify directories and a plurality of file types that are to be accessed in the remote data store, wherein the remote data store stores heterogeneous data in files having different file types;
- allocate a plurality of processors to communicate with the remote data store;
- generate instructions for worker nodes, wherein the generated instructions include instructions to:
- scan, using the plurality of processors, the identified directories;
- open, using a processor of the plurality of processors, a file in the remote data store, the file having a file type of the plurality of file types identified in the configuration file;
- apply, using the processor, filter criteria to content of the file;
- and
- include, using the processor, data from the file that satisfies the filter criteria in a partition, wherein the plurality of processors concurrently process a plurality of partitions including the partition, wherein each partition comprises data from one or more files of the remote data store;
- and
- execute a query based on the generated instructions.
Non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:
- receive a query that references a remote data store;
- access a configuration file to identify directories and a plurality of file types that are to be accessed in the remote data store, wherein the remote data store stores heterogeneous data in files having different file types;
- allocate a plurality of processors to communicate with the remote data store;
- generate instructions for worker nodes, wherein the generated instructions include instructions to:
- scan, using the plurality of processors, the identified directories;
- open, using a processor of the plurality of processors, a file in the remote data store, the file having a file type of the plurality of file types identified in the configuration file;
- apply, using the processor, filter criteria to content of the file;
- and
- include, using the processor, data from the file that satisfies the filter criteria in a partition, wherein the plurality of processors concurrently process a plurality of partitions including the partition, wherein each partition comprises data from one or more files of the remote data store;
- and execute a query based on the generated instructions.
Dependencies are inferred from the claim text.