System and method for visualizing dynamic diagrams representing data streaming system configuration

A system for visualizing a real-time configuration change of a data streaming system, comprising:

• one or more processors;
• and memory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
• displaying a graphic user interface (GUI) representing a data streaming system, wherein:
• the data streaming system comprises a data storage engine and a data processing engine, the data storage engine comprises a configurable number of partitions for ingesting a plurality of data streams in parallel, the data processing engine is configured to fetch data from the partitions for parallel processing, and the GUI comprises a first displaying area and a second displaying area, wherein:
• the first displaying area displays (1) a plurality of channels respectively corresponding to the partitions, and (2) a plurality of dynamically moving objects moving through the plurality of channels representing data being streamed into the corresponding partitions, and the second displaying area displays a user-configuration interface allowing a user to adjust one or more configurations of the data streaming system;
• receiving an adjusted configuration from the user-configuration interface;
• automatically adjusting the data storage engine and/or the data processing engine based on the adjusted configuration;
• dynamically scaling the plurality of channels displayed in the first displaying area to reflect a change of the number of partitions in response to the adjusted configuration;
• and dynamically increasing or decreasing a moving speed of the moving objects moving through the plurality of channels to reflect a change of processing speed in response to the adjusted configuration, wherein:
• the user-configuration interface is configured for a user to adjust the number of partitions in the data storage engine.

A non-transitory computer readable media comprising instructions that, when executed, cause one or more processors to perform operations comprising:

• displaying a graphic user interface (GUI) representing a data streaming system, wherein:
• the data streaming system comprises a data storage engine and a data processing engine, the data storage engine comprises a configurable number of partitions for ingesting a plurality of data streams in parallel, the data processing engine is configured to fetch data from the partitions for parallel processing, and the GUI comprises a first displaying area and a second displaying area, wherein:
• the first displaying area displays (1) a plurality of channels respectively corresponding to the partitions, and (2) a plurality of dynamically moving objects moving through the plurality of channels representing data being streamed into the corresponding partitions, and the second displaying area displays a user-configuration interface allowing a user to adjust one or more configurations of the data streaming system;
• receiving an adjusted configuration from the user-configuration interface;
• automatically adjusting the data storage engine and/or the data processing engine based on the adjusted configuration;
• dynamically scaling the plurality of channels displayed in the first displaying area to reflect a change of the number of partitions in response to the adjusted configuration;
• and dynamically increasing or decreasing a moving speed of the moving objects moving through the plurality of channels to reflect a change of processing speed in response to the adjusted configuration, wherein the one or more configurations comprise a throughput configuration, a key configuration, or a partition configuration.