# US 11,442,935 B2 · Determining a record generation estimate of a processing task > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,442,935 B2 - Document ID: US11442935B2 - Status: Granted - Kind: B2 - Application number: 16/397,930 - Filed: 2019-04-29 - Published: 2019-08-22 - Granted: 2022-09-13 - Earliest claimed: 2016-09-26 - Assignees: Splunk Inc. - Applicants: Splunk Inc. - Inventors: Sourav Pal; Arindam Bhattacharjee; Asha Andrade - Web page: https://patentgrep.com/patent/US11442935B2 ## CPC - G06F 16/24542: {Plan optimisation} - G06F 9/505: {considering the load} - G06F 16/2453: Query optimisation - G06F 16/2455: Query execution - G06F 16/2458: Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries - G06F 16/2465: {Query processing support for facilitating data mining operations in structured databases} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Systems and methods are described for determining a record generation estimate related to a particular processing task. The system obtains a sample set of data that includes multiple records. The system applies a processing task, such as a transform or regular expression rule to the sample set of data and determines how many records are generated by the processing task. Based on the number of records generated, the system determines a record generation estimate. The system can use the record generation estimate to allocate compute resources or determine a query execution time for at least a portion of the query based on the record generation estimate. ## Claims ### Claim 1 A method, comprising: receiving a search query, the search query identifying a set of data to be processed and a manner of processing the set of data; generating a search query execution scheme to execute the search query in a distributed computing execution environment, wherein generating the search query execution scheme comprises: obtaining a sample set of data of the set of data to be processed, wherein the sample set of data comprises a plurality of records; applying a processing task to the plurality of records; determining a quantity of records generated based on the applying the processing task; determining a record generation estimate based on a ratio of the quantity of records generated and a quantity of the plurality of records; storing the record generation estimate in association with the processing task; allocating compute resources for execution of at least a portion of the search query that includes the processing task based on the record generation estimate; and executing the search query based at least in part on the search query execution scheme and the allocated compute resources, wherein executing the search query comprises identifying the set of data to be processed and processing the set of data in the manner indicated by the search query using at least the allocated compute resources. ### Claim 2 (depends on 1) The method of claim 1, wherein the sample set of data is obtained from a bucket of a data intake and query system. ### Claim 3 (depends on 1) The method of claim 1, wherein the sample set of data corresponds to a chunk of data obtained from a bucket of a data intake and query system. ### Claim 4 (depends on 1) The method of claim 1, wherein the sample set of data is obtained based on the processing task. ### Claim 5 (depends on 1) The method of claim 1, wherein the sample set of data is obtained based on the processing task, wherein the processing task identifies at least one of an index or sourcetype used to obtain records for the sample set of data. ### Claim 6 (depends on 1) The method of claim 1, wherein the sample set of data is obtained based on an identified index and time range of events stored in a data store, wherein each event of the events includes a portion of raw machine data associated with a timestamp. ### Claim 7 (depends on 1) The method of claim 1, wherein the plurality of records are based on events stored in a data store, wherein each event of the events includes a portion of raw machine data associated with a timestamp. ### Claim 8 (depends on 1) The method of claim 1, wherein the processing task is an extraction rule. ### Claim 9 (depends on 1) The method of claim 1, wherein the processing task is a data transform. ### Claim 10 (depends on 1) The method of claim 1, further comprising allocating and deallocating the compute resources at different stages of the search query for different portions of the search query. ### Claim 11 (depends on 1) The method of claim 1, wherein the processing task is a first processing task, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, the method further comprising: applying a second processing task to the plurality of records; determining a second quantity of records generated based on the applying the second processing task; determining a second record generation estimate based on the second quantity of records generated; and storing the second record generation estimate with the first record generation estimate. ### Claim 12 (depends on 1) The method of claim 1, wherein the sample set of data is a first sample set of data, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, the method further comprising: applying the processing task to a second sample set of data; determining a second quantity of records generated based on the applying the processing task to the second sample set of data; determining a second record generation estimate based on the second quantity of records generated; and storing the second record generation estimate with the first record generation estimate. ### Claim 13 (depends on 1) The method of claim 1, wherein the sample set of data is a first sample set of data associated with a first time range of data, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, the method further comprising: applying the processing task to a second sample set of data associated with a second time range of data; determining a second quantity of records generated based on the applying the processing task to the second sample set of data; determining a second record generation estimate based on the second quantity of records generated; and storing the second record generation estimate with the first record generation estimate. ### Claim 14 (depends on 1) The method of claim 1, wherein the obtaining the sample set of data is based on an addition of the processing task to a data intake and query system. ### Claim 15 (depends on 1) The method of claim 1, wherein the obtaining the sample set of data is based on an addition of one or more events corresponding to the sample set of data to a data intake and query system. ### Claim 16 (depends on 1) The method of claim 1, wherein the record generation estimate is stored in a configuration file that stores a plurality of record generation estimates, each record generation estimate corresponding to a unique combination of a sample set of data and a processing task. ### Claim 17 (depends on 1) The method of claim 1, wherein the record generation estimate is stored in a configuration file that stores a plurality of record generation estimates, wherein the configuration file stores a time range and processing task identifier for each record generation estimate of the plurality of record generation estimates. ### Claim 18 (depends on 1) The method of claim 1, wherein the record generation estimate is stored in a configuration file that stores a plurality of record generation estimates, wherein the configuration file stores a time range, processing task identifier, and sourcetype identifier for each record generation estimate of the plurality of record generation estimates. ### Claim 19 (depends on 1) The method of claim 1, wherein the at least the portion of the search query includes the processing task, and further comprising allocating and deallocating the compute resources at different stages of the search query for different portions of the search query. ### Claim 20 (depends on 1) The method of claim 1, wherein the processing task is a first processing task, and further comprising determining a number of records generated by a first processing task; and determining a number of records generated by a subsequent task based on the number of records generated by the first processing task and a subsequent task record generation estimate. ### Claim 21 (depends on 1) The method of claim 1, wherein allocating the compute resources comprises allocating a quantity of worker nodes to execute the at least the portion of the search query based on the record generation estimate. ### Claim 22 (depends on 1) The method of claim 1, wherein allocating the compute resources comprises allocating a quantity of processors of one or more worker nodes to execute the at least a portion of the search query based on the record generation estimate. ### Claim 23 (depends on 1) The method of claim 1, further comprising estimating a processing time for the at least the portion of the search query. ### Claim 24 (depends on 1) The method of claim 1, further comprising estimating a processing time for the at least the portion of the search query based on the record generation estimate and the compute resources allocated for the at least the portion of the search query. ### Claim 25 A computing system of a data intake and query system, the computing system comprising: memory; and one or more processing devices coupled to the memory and configured to: receiving a search query, the search query identifying a set of data to be processed and a manner of processing the set of data; generating a search query execution scheme to execute the search query in a distributed computing execution environment, wherein generating the search query execution scheme comprises: obtain a sample set of data of the set of data to be processed, wherein the sample set of data comprises a plurality of records; apply a processing task to the plurality of records; determine a quantity of records generated based on application of the processing task to the plurality of records; determine a record generation estimate based on a ratio of the quantity of records generated and a quantity of the plurality of records; storing the record generation estimate in association with the processing task; allocate compute resources for execution of at least a portion of the search query that includes the processing task based on the record generation estimate; and executing the search query based at least in part on the search query execution scheme and the allocated compute resources, wherein executing the search query comprises identifying the set of data to be processed and processing the set of data in the manner indicated by the search query using at least the allocated compute resources. ### Claim 26 (depends on 25) The system of claim 25, wherein the sample set of data is a first sample set of data, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, and wherein the one or more processing devices are further configured to: apply the processing task to a second sample set of data; determine a second quantity of records generated based on the applying the processing task to the second sample set of data; determine a second record generation estimate based on the second quantity of records generated; and store the second record generation estimate with the first record generation estimate. ### Claim 27 (depends on 25) The system of claim 25, wherein the sample set of data is a first sample set of data associated with a first time range of data, the quantity of records generated is a first quantity of records generated, and the record generation estimate is a first record generation estimate, and wherein the one or more processing devices are further configured to: apply the processing task to a second sample set of data associated with a second time range of data; determine a second quantity of records generated based on the applying the processing task to the second sample set of data; determine a second record generation estimate based on the second quantity of records generated; and store the second record generation estimate with the first record generation estimate. ### Claim 28 Non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system of a data intake and query system, cause the computing system to: receiving a search query, the search query identifying a set of data to be processed and a manner of processing the set of data; generating a search query execution scheme to execute the search query in a distributed computing execution environment, wherein generating the search query execution scheme comprises: obtain a sample set of data of the set of data to be processed, wherein the sample set of data comprises a plurality of records; apply a processing task to the plurality of records; determine a quantity of records generated based on application of the processing task to the plurality of records; determine a record generation estimate based on a ratio of the quantity of records generated and a quantity of the plurality of records; storing the record generation estimate in association with the processing task; allocate compute resources for execution of at least a portion of the search query that includes the processing task based on the record generation estimate; and executing the search query based at least in part on the search query execution scheme and the allocated compute resources, wherein executing the search query comprises identifying the set of data to be processed and processing the set of data in the manner indicated by the search query using at least the allocated compute resources. ### Claim 29 (depends on 28) The non-transitory computer-readable media of claim 28, wherein the record generation estimate is stored in a configuration file that stores a plurality of record generation estimates, each record generation estimate corresponding to a unique combination of a sample set of data and a processing task. ### Claim 30 (depends on 28) The non-transitory computer-readable media of claim 28, wherein the record generation estimate is stored in a configuration file that stores a plurality of record generation estimates, wherein the configuration file stores a time range and processing task identifier for each record generation estimate of the plurality of record generation estimates.