US 9,807,192 B1 · Centralized coordination of data collection tasks from multiple sources

Abstract

A scheduler manages execution of a plurality of data-collection jobs, assigns individual jobs to specific forwarders in a set of forwarders, and generates and transmits tokens (e.g., pairs of data-collection tasks and target sources) to assigned forwarders. The forwarder uses the tokens, along with stored information applicable across jobs, to collect data from the target source and forward it onto an indexer for processing. For example, the indexer can then break a data stream into discrete events, extract a timestamp from each event and index (e.g., store) the event based on the timestamp. The scheduler can monitor forwarders' job performance, such that it can use the performance to influence subsequent job assignments. Thus, data-collection jobs can be efficiently assigned to and executed by a group of forwarders, where the group can potentially be diverse and dynamic in size.

Claims30

Dependent claims

A computer-implemented method comprising:

identifying a plurality of first jobs, each first job in the plurality of first jobs including collecting data from a source;
assigning each first job in the plurality of first jobs to a forwarder in a set of forwarders, each forwarder in the set of forwarders being configured to collect data from a source and to transmit the collected data to an indexer to be processed by the indexer and stored;
transmitting, for each first job in the plurality of first jobs, data indicative of the first job to the forwarder assigned to the first job;
receiving, from each of one or more forwarders in the set of forwarders, a communication indicative of a completion status of the assigned first job;
identifying a second job that includes collecting data;
selecting a forwarder of the set of forwarders to perform the second job, based on the received communication indicative of the completion status of the assigned first job;
and
assigning the second job to the selected forwarder.

5 more elements

A system comprising:

a communication interface through which the system can communicate with a remote entity via a network;
and
one or more data processors coupled to the communication interface and individually or collectively programmed to perform operations including:
identifying a plurality of first jobs, wherein each first job in the plurality of first jobs includes collecting data from a source,
assigning each first job in the plurality of first jobs to a forwarder in a set of forwarders, each forwarder in the set of forwarders being configured to collect data from a source and transmit the collected data to an indexer to be processed by the indexer and stored,
transmitting, for each first job in the plurality of first jobs, data indicative of the first job to the forwarder assigned to the first job,
receiving a communication from each of one or more forwarders in the set of forwarders, wherein the communication is indicative of a completion status of the assigned first job,
identifying a second job, wherein the second job includes collecting data,
selecting a forwarder of the set of forwarders to perform the second job, based on the received communication indicative of the completion status of the assigned first job, and
assigning the second job to the selected forwarder.

7 more elements

A non-transitory machine-readable medium storing instructions, execution of which in a processing system causes one or more processors in the processing system to:

identify a plurality of first jobs, wherein each first job in the plurality of first jobs includes collecting data from a source;
assign each first job in the plurality of first jobs to a forwarder in a set of forwarders, each forwarder in the set of forwarders being configured to collect data from a source and transmit the collected data to an indexer to be processed by the indexer and stored;
transmit, for each first job in the plurality of first jobs, data indicative of the first job, wherein the data is transmitted to the forwarder assigned to the first job;
receive a communication from each of one or more forwarders in the set of forwarders, wherein the communication is indicative of a completion status of the assigned first job;
identify a second job that includes collecting data;
select a forwarder of the set of forwarders to perform the second job, based on the received communication indicative of the completion status of the assigned first job;
and
assign the second job to the selected forwarder.

5 more elements

Dependencies are inferred from the claim text.

Centralized coordination of data collection tasks from multiple sources

Life of this patent

Centralized coordination of data collection tasks from multiple sources

Life of this patent

Abstract

Claims30