Comprehensive telemetry data management in cluster networks with dynamic dataset registration and processing

A system for processing telemetry data in a cluster network having a plurality of nodes, comprising:

• a telemetry collector receiving telemetry data from a plurality of telemetry producers, wherein the received telemetry data is formatted into a structured format for storage in a central datastore, wherein the telemetry data can be previously generated data or new data;
• a telemetry transmitter validating one or more consumers of respective datasets of the telemetry data in the network, wherein the one or more consumers subscribe to receive the respective data through a subscription process;
• and
• a pipeline transmitting the respective data to subscribed consumers through a selected transport mechanism, wherein the telemetry data comprises data generated periodically by each producer upon operation in the cluster network, and wherein the telemetry data comprises performance data, topology information, alerts, security states, and service features, and can comprise discrete datasets or streaming telemetry data, and further wherein the one or more consumers comprises at least one of:
• pod components of the nodes, storage users, graphical user interfaces (GUI), and storage vendors, and wherein the collector receives a schedule to receive the telemetry data on an individual basis from one or more consumers of respective data of the telemetry data in the network, and transmits the respective data to the one or more consumers through a selected transport mechanism and at a respective frequency based on the received schedule.

A system for processing telemetry data in a cluster network having a plurality of nodes, comprising:

• a telemetry collector registering producers and consumers based on defined schema of the telemetry data;
• an access control component restricting access to the telemetry data based on role-based access controls (RBAC);
• a security control component restrict production and use based on compliance regulations regarding security and restricted use of the telemetry data;
• a self-healing component detecting generation of telemetry data in excess of defined thresholds of normal operation for one or more defined golden signal metric datasets;
• an optimizer component optimizing processing of streaming time-series telemetry data with respect to data storage and network bandwidth usage;
• and
• a datastore storing a telemetry catalog accessed by the telemetry collector and storing schema definitions for new and existing telemetry data for registering the producers, consumers, and telemetry datasets;
• and
• a pipeline transmitting the telemetry datasets to validated consumers through selected transport mechanisms.