Data forwarder for distributed data acquisition, indexing and search system

An apparatus to implement a forwarder for a data acquisition, indexing and search system, the apparatus comprising:

• a memory;
• a network interface through which to communicate with at least one other device in a network environment that includes a plurality of forwarders configured to collect data from a respective plurality of data sources, and a plurality of indexers configured to index data collected by the plurality of forwarders for subsequent searching of indexed data;
• and
• a processor coupled to the memory and the network interface and configured to execute operations including
• receiving credentials, by the forwarder via the network;
• storing the credentials, by the forwarder;
• maintaining a plurality of worker processes in the forwarder;
• transmitting, by the forwarder to a scheduler, capabilities information including an indication of a number of the worker processes in the forwarder that can execute jobs in parallel;
• storing information relating to a plurality of jobs in a queue, each of the plurality of worker processes configured to execute at least one task from the queue;
• receiving job configuration data, by the forwarder, the job configuration data including indications of a first job including a data collection task for the forwarder to perform, a target source from which data is to be collected in the data collection task, and a particular indexer to which the forwarder is to forward the data after collection of the data from the target source, the first job having been assigned to the forwarder by the scheduler based on the indication of the number of the worker processes in the forwarder that can execute jobs in parallel;
• using, by the particular forwarder, the job configuration data and the stored credentials to execute the first job, including to collect data from the target source, wherein the data collected from the target source includes unstructured machine data;
• and
• transmitting, by the particular forwarder to the particular indexer, data collected from the target source according to the first job, for indexing of the data collected from the target source.

A non-transitory machine-readable storage medium having stored therein instructions, execution of which in a processing device causes the processing device to perform operations of a forwarder in a network environment that includes a plurality of forwarders configured to collect data from a respective plurality of data sources, and a plurality of indexers configured to index data collected by the plurality of forwarders for subsequent searching of indexed data, the operations of the forwarder comprising:

• receiving credentials via a network;
• storing the credentials;
• maintaining a plurality of worker processes in the forwarder;
• transmitting, by the forwarder to a scheduler, capabilities information including an indication of a number of the worker processes in the forwarder that can execute jobs in parallel;
• storing information relating to a plurality of jobs in a queue, each of the plurality of worker processes configured to execute at least one task from the queue;
• receiving job configuration data, the job configuration data including indications of a first job including a data collection task for the forwarder to perform, a target source from which data is to be collected in the data collection task, and a particular indexer to which the forwarder is to forward the data after collection of the data from the target source, the first job having been assigned to the forwarder by the scheduler based on the indication of the number of the worker processes in the forwarder that can execute jobs in parallel;
• using the job configuration data and the stored credentials to execute the first job, including to collect data from the target source, wherein the data collected from the target source includes unstructured machine data;
• and
• transmitting, to the particular indexer, data collected from the target source according to the first job, for indexing of the data collected from the target source.