Distributed data analysis for streaming data sources

A server system configured to analyze data for a plurality of machines located at a non-static collection of nodes that form a first linear communication orbit, comprising:

• one or more processors;
• and
• memory storing a plurality of function modules for execution by the one or more processors, each function module configured to receive data related to a core function of the function module from a plurality of machines, distinct from the server system, each respective function module including an internal client configured to perform a set of operations including:
• receiving a data collection instruction, wherein the data collection instruction includes a schema definition, the schema definition defining distinct first and second data types for data derived from data, of one or more third data types, to be collected from nodes in one or more linear communication orbits, including the first linear communication orbit, and a data structure for organizing data items of the first and second data types;
• converting the data collection instruction, including the schema definition, to a data collection request for collecting a plurality of raw data items from a subset of the plurality of machines in the first linear communication orbit according to the schema definition, including identifying each of the plurality of raw data items based on the schema definition, wherein the plurality of raw data items includes the one or more third data types;
• sending the data collection request through the first linear communication orbit to the subset of the plurality of machines, wherein the data collection request propagates from node to node along the first linear communication orbit until reaching each machine in the subset of the plurality of machines;
• in response to the data collection request, collecting the plurality of raw data items from the subset of the plurality of machines via the first linear communication orbit;
• analyzing the plurality of raw data items collected from the subset of the plurality of machines, including:
• in accordance with the schema definition, obtaining a first subset of raw data items that have the one or more third data types, deriving a set of data items having the first data type and second data type based on the first subset of raw data items, and organizing the set of derived data items according to the data structure defined in the schema definition, wherein result data of the analysis include the set of derived data items;
• in accordance with the analysis of the raw data items, storing the result data in a database according to the schema definition.

A non-transitory computer-readable storage medium, having one or more programs stored thereon, which when executed by one or more processors of a server system, cause the server system to perform operations for analyzing data for a plurality of machines located at a non-static collection of nodes that form a first linear communication orbit, wherein the one or more programs include a plurality of function modules, each function module configured to receive data related to a core function of the function module from a plurality of machines, distinct from the server system, each respective function module including an internal client configured to perform a set of operations comprising:

• receiving a data collection instruction, wherein the data collection instruction includes a schema definition, the schema definition defining distinct first and second data types for data derived from data, of one or more third data types, to be collected from nodes in one or more linear communication orbits, including the first linear communication orbit, and a data structure for organizing data items of the first and second data types;
• converting the data collection instruction, including the schema definition, to a data collection request for collecting a plurality of raw data items from a subset of the plurality of machines in the first linear communication orbit according to the schema definition, including identifying each of the plurality of raw data items based on the schema definition, wherein the plurality of raw data items includes the one or more third data types;
• sending the data collection request through the first linear communication orbit to the subset of the plurality of machines, wherein the data collection request propagates from node to node along the first linear communication orbit until reaching each machine in the subset of the plurality of machines;
• in response to the data collection request, collecting the plurality of raw data items from the subset of the plurality of machines via the first linear communication orbit;
• analyzing the plurality of raw data items collected from the subset of the plurality of machines, including:
• in accordance with the schema definition, obtaining a first subset of raw data items that have the one or more third data types, deriving a set of data items having the first data type and second data type based on the first subset of raw data items, and organizing the set of derived data items according to the data structure defined in the schema definition, wherein result data of the analysis include the set of derived data items;
• in accordance with the analysis of the raw data items, storing the result data in a database according to the schema definition.