Data Fabric Service System Architecture

The method of claim 10, wherein defining the search scheme comprises:

• determining, by the data intake and query system, that the search query requires searching the plurality of distributed data storage systems;
• and
• defining a plurality of search phases for searching the plurality of distributed data storage systems and obtaining the search results from the search service;
• wherein the search scheme includes the plurality of search phases, and the portion of the search scheme includes at least one of the plurality of search phases.

The method of claim 10, wherein defining the search scheme comprises:

• determining, by the data intake and query system, that the search query requires searching the plurality of distributed data storage systems;
• and
• defining a plurality of ordered search phases for searching the plurality of distributed data storage systems and obtaining the search results from the search service in accordance with an order of the plurality of ordered search phases;
• wherein the search scheme includes the plurality of ordered search phases, and the portion of the search scheme includes at least one of the plurality of ordered search phases for searching the external data storage system after execution of another one of the plurality of ordered search phases for searching the internal data storage system.

A method comprising:

• receiving, by a data intake and query system, a search query;
• defining, by the data intake and query system, a search scheme for applying the search query on a plurality of distributed data storage systems including an internal data storage system of the data intake and query system and an external data storage system communicatively coupled to the data intake and query system over a computer network, wherein the internal data storage system stores data as a plurality of time-indexed events including respective segments of raw machine data;
• transferring, by the data intake and query system, a portion of the search scheme to a search service for obtaining partial search results from the external data storage system;
• producing, by the data intake and query system, partial search results by applying a portion of the search scheme to the internal data storage system;
• transmitting, by the data intake and query system over the computer network, the partial search results to at least one worker node communicatively coupled to the external data storage system over the computer network, the at least one worker node being communicatively coupled to the search service;
• receiving, by the data intake and query system from the search service, search results of the search query obtained by application of the search scheme to the plurality of distributed data storage systems including the internal data storage system and the external data storage system;
• wherein the search results obtained by the data intake and query system from the search service are based on a combination of the partial search results of the internal data storage system and the partial search results obtained from the external data storage system combined by the at least one worker node;
• and
• causing output of the search results or data indicative of the search results by a display device to a user.

A method comprising:

• receiving, by a search service, data representing a portion of a search scheme defined by a data intake and query system for a plurality of worker nodes to collect a plurality of partial search results over a computer network from a plurality of distributed data storage systems including a plurality of external data storage systems and an internal data storage system of the data intake and query system, wherein the search scheme comprises a plurality of search phases;
• defining, by the search service, an executable search process based on the received portion of the search scheme defined by the data intake and query system, the executable search process being defined as a directed acyclic graph to include a first phase causing the plurality of worker nodes to extract and collect the plurality of partial search results from the plurality of external data storage systems, a second phase causing the plurality of worker nodes to collect the plurality of partial search results from the internal data storage system, and a third phase causing the search service to collect the plurality of partial search results of the plurality of external data storage systems and the plurality of partial search results of the internal data storage system from the plurality of worker nodes;
• executing, by the search service, the executable search process to cause the plurality of worker nodes to extract and collect the plurality of partial search results from the plurality of external data storage systems and collect the plurality of partial search results from the internal data storage system over the computer network;
• receiving, by the search service, a plurality of aggregate partial search results based on the plurality of partial search results of the plurality of external data storage systems and the plurality of partial search results of the internal data storage system;
• and
• sending, by the search service, the plurality of aggregate partial search results to the data intake and query system.

A method comprising:

• receiving, by a search service, data representing a portion of a search scheme defined by a data intake and query system for a plurality of worker nodes to collect partial search results over a computer network from a plurality of distributed data storage systems including a plurality of external data storage systems and an internal data storage system of the data intake and query system;
• defining, by the search service, an executable search process based on the received portion of the search scheme defined by the data intake and query system;
• executing, by the search service, the executable search process to cause the plurality of worker nodes to collect a plurality of partial search results from the plurality of external data storage systems and collect a plurality of partial search results from the internal data storage system over the computer network and produce a plurality of aggregate partial search results by combining the plurality of partial search results of the plurality of external data storage systems with the plurality of partial search results of the internal data storage system as an arrangement of time-ordered events;
• receiving, by the search service, the plurality of aggregate partial search results;
• and
• sending, by the search service, the plurality of aggregate partial search results to the data intake and query system.

A method comprising:

• receiving, by a worker node over a computer network, search instructions defined by a search service for the worker node to collect a plurality of partial search results from a plurality of distributed data storage systems including a plurality of partial search results of an external data storage system and a plurality of partial search results of an internal data storage system of a data intake and query system;
• collecting, by the worker node over a computer network, the plurality of partial search results of the external data storage system searched in accordance with the search instructions;
• collecting, by the worker node over the computer network, the plurality of partial search results of the internal data storage system;
• producing, by the worker node, a plurality of aggregate partial search results from the plurality of partial search results of the external data storage system and the plurality of partial search results of the internal data storage system by combining the plurality of partial search results of the external data storage system with the plurality of partial search results of the internal data storage system as an arrangement of time-ordered events;
• and
• providing, by the worker node over the computer network to the search service, the plurality of aggregate partial search results in response to the search instructions.

A method comprising:

• receiving, by a first worker node over a computer network, search instructions defined by a search service for the first worker node to collect a plurality of partial search results from a plurality of distributed data storage systems including a first plurality of partial search results of a first external data storage system and a second plurality of partial search results of an internal data storage system of a data intake and query system;
• collecting, by the first worker node over a computer network, the first plurality of partial search results of the first external data storage system searched in accordance with the search instructions;
• collecting, by the first worker node over the computer network, the second plurality of partial search results of the internal data storage system;
• collecting, by the first worker node over the computer network, a third plurality of partial search results extracted from a second external data storage system by a second worker node, the second worker node being communicatively coupled over the computer network to the second external data storage system;
• sending, by the first worker node over the computer network, at least a portion of the first plurality of partial search results to the second worker node;
• producing, by the first worker node, a plurality of aggregate partial search results from the first plurality of partial search results of the first external data storage system, the second plurality of partial search results of the internal data storage system, and the third plurality of partial search results of the second external data storage system;
• and
• providing, by the first worker node over the computer network to the search service, the plurality of aggregate partial search results in response to the search instructions.