Achieve search and ingest isolation via resource management in a search and indexing system

A system comprising:

• a plurality of hardware resources comprising a first hardware resource and a second hardware resource;
• a plurality of workload pools for the plurality of hardware resources, wherein each of the plurality of workload pools is a non-overlapping partition of the plurality of hardware resources,
• wherein the plurality of workload pools is defined according to a workload pool hierarchy comprising a plurality of workload pool layers, the plurality of workload pool layers comprising:
• a first layer comprising a node for allocating a set of workload pools of the plurality of workload pools to data intake and query system processes,
• a second layer partitioning the set of workload pools for data intake and query system processes into a first subset allocated for query processes, a second subset allocated for ingest processes, and a third subset allocated for user input processes, wherein the data intake and query system processes comprise the query processes, the ingest processes, and the user input processes, wherein the first subset, the second subset, and the third subset are disjoint, the first subset comprising the first hardware resource and the second subset comprising the second hardware resource, wherein the ingest processes comprise processes to ingest data into a data intake and query system, the query processes comprise processes to query the data in the data intake and query system, and the user input processes comprise processes to monitor and manage the data intake and query system, and
• a third layer partitioning the first subset, the second subset, and the third subset into individual workload pools, wherein the individual workload pools are in leaf nodes of the workload pool hierarchy;
• and
• one or more computer processors configured to:
• receive a processing request,
• assign the processing request to one workload pool selected from a group consisting of the first subset and the second subset to obtain a selected workload pool, wherein assigning the processing request to one workload pool comprises selecting, from the individual workload pools, in the third layer, the workload pool that is allocated to a subset based on the processing request corresponding to one selected from a group consisting of an ingest process and a query process;
• and
• initiate execution of the processing request on the selected workload pool.

A non-transitory computer readable medium comprising computer readable program code for performing operations, the operations comprising:

• receiving a search query associated with at least one data store;
• assigning, in response to the search query being associated with the at least one data store, the search query to a first workload pool in a first subset of workload pools of a plurality of workload pools,
• wherein each of the plurality of workload pools is a non-overlapping partition of hardware resources,
• wherein the plurality of workload pools is defined according to a workload pool hierarchy comprising a plurality of workload pool layers, the plurality of workload pool layers comprising:
• a first layer comprising a node for allocating a set of workload pools of the plurality of workload pools to data intake and query system processes,
• a second layer partitioning the set of workload pools for data intake and query system processes into the first subset allocated for query processes, a second subset allocated for ingest processes, and a third subset allocated for user input processes, wherein the data intake and query system processes comprise the query processes, the ingest processes, and the user input processes, wherein the first subset, the second subset, and the third subset are disjoint, and wherein the ingest processes comprise processes to ingest data into a data intake and query system, the query processes comprise processes to query the data in the data intake and query system, and the user input processes comprise processes to monitor and manage the data intake and query system, and
• a third layer partitioning the first subset, the second subset, and the third subset into individual workload pools, wherein the individual workload pools are in leaf nodes of the workload pool hierarchy, and
• wherein assigning the search query to the first workload pool in the first subset comprises selecting the first workload pool from the individual workload pools in the third layer that are allocated to the first subset, based on the search query corresponding to a query process;
• processing the search query using a first hardware resource in the first workload pool;
• receiving an ingest request comprising data associated with the at least one data store, wherein the ingest request is assigned to a second workload pool in the second subset, wherein assigning the ingest request to the second workload pool in the second subset comprises selecting the second workload pool from the individual workload pools in the third layer that are allocated to the second subset based on the ingest request corresponding to an ingest process;
• and
• processing the ingest request using a second hardware resource in the second workload pool.