Determining a record generation estimate of a processing task

A computing system of a data intake and query system, the computing system comprising:

• memory;
• and
• one or more processing devices coupled to the memory and configured to:
• receiving a search query, the search query identifying a set of data to be processed and a manner of processing the set of data;
• generating a search query execution scheme to execute the search query in a distributed computing execution environment, wherein generating the search query execution scheme comprises:
• obtain a sample set of data of the set of data to be processed, wherein the sample set of data comprises a plurality of records;
• apply a processing task to the plurality of records;
• determine a quantity of records generated based on application of the processing task to the plurality of records;
• determine a record generation estimate based on a ratio of the quantity of records generated and a quantity of the plurality of records;
• storing the record generation estimate in association with the processing task;
• allocate compute resources for execution of at least a portion of the search query that includes the processing task based on the record generation estimate;
• and
• executing the search query based at least in part on the search query execution scheme and the allocated compute resources, wherein executing the search query comprises identifying the set of data to be processed and processing the set of data in the manner indicated by the search query using at least the allocated compute resources.

Non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to:

• receiving a search query, the search query identifying a set of data to be processed and a manner of processing the set of data;
• generating a search query execution scheme to execute the search query in a distributed computing execution environment, wherein generating the search query execution scheme comprises:
• obtain a sample set of data of the set of data to be processed, wherein the sample set of data comprises a plurality of records;
• apply a processing task to the plurality of records;
• determine a quantity of records generated based on application of the processing task to the plurality of records;
• determine a record generation estimate based on a ratio of the quantity of records generated and a quantity of the plurality of records;
• storing the record generation estimate in association with the processing task;
• allocate compute resources for execution of at least a portion of the search query that includes the processing task based on the record generation estimate;
• and
• executing the search query based at least in part on the search query execution scheme and the allocated compute resources, wherein executing the search query comprises identifying the set of data to be processed and processing the set of data in the manner indicated by the search query using at least the allocated compute resources.