System and method for optimization and load balancing of computer clusters

A system for optimization and load balancing for computer clusters, comprising:

• a distributed computational graph module comprising a first plurality of programming instructions stored in the memory and operating on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the processor to:
• receive a distributed computational graph defining a data processing workflow wherein:
• the vertices of the distributed computational graph represent data transformation stages and the edges of the distributed computational graph represent messaging between the data transformation stages;
• and
• the data processing workflow comprises one or more data pipelines for analysis of load balancing of a plurality of devices, each data pipeline comprising a series of nodes and edges of the directed computational graph;
• maintain a plurality of connections with each of the plurality of devices over a network, wherein each connection provides the ability to send data to, and receive data from, the respective device over the network;
• and
• analyze a dataset using the data pipeline as requested by a load forecasting application to produce a load balancing result;
• the load forecasting application comprising a second plurality of programming instructions stored in the memory and operating on the processor, wherein the second plurality of programming instructions, when operating on the processor, causes the processor to:
• query a multidimensional time-series database for portions of the recorded data for one or more of the plurality of devices;
• utilize the received response data to produce a load simulation, wherein the load simulation comprises a the distributed computational graph;
• provide the distributed computational graph to the directed computational graph module for execution;
• receive the load balancing result from the directed computational graph;
• and
• create a redistribution of a processing load among the plurality of devices.