Method, system, and storage medium of machine-learningbased real-time task scheduling for apache storm cluster

A system, comprising:

• a memory, configured to store program instructions for performing a machine-learning-based real-time task scheduling method, applied to an Apache Storm cluster, wherein the Apache Storm cluster includes a master node and a plurality of worker nodes;
• and
• a processor, coupled with the memory and, when executing the program instructions, configured for:
• for a worker node of the plurality of worker nodes, executing a training task distributed by the master node;
• collecting latency time lengths of each of a plurality of machine learning models under different CPU (central processing unit) utilization and memory usage;
• calculating a mean squared error of the latency time lengths of each of the plurality of machine learning models;
• comparing the plurality of machine learning models according to mean squared errors of latency time lengths corresponding to the plurality of machine learning models to select a desirable machine learning model;
• and installing the desirable machine learning model on the worker node;
• providing an API (application programming interface) for the worker node configured for communication between the master node and the worker node;
• when receiving a task by the master node, requesting the worker node to predict a latency time length according to current CPU utilization and current memory usage of the worker node;
• and returning the predicted latency time length to the master via the API of the worker node;
• and
• after the master node collects predicted latency time lengths of the plurality of worker nodes, assigning the task to a corresponding worker node with a lowest predicted latency time length.

A non-transitory computer-readable storage medium, containing program instructions for, when being executed by a processor, performing a machine-learning-based real-time task scheduling method, applied to an Apache Storm cluster, wherein the Apache Storm cluster includes a master node and a plurality of worker nodes;

• the method comprising:
• for a worker node of the plurality of worker nodes, executing a training task distributed by the master node;
• collecting latency time lengths of each of a plurality of machine learning models under different CPU (central processing unit) utilization and memory usage;
• calculating a mean squared error of the latency time lengths of each of the plurality of machine learning models;
• comparing the plurality of machine learning models according to mean squared errors of latency time lengths corresponding to the plurality of machine learning models to select a desirable machine learning model;
• and installing the desirable machine learning model on the worker node;
• providing an API (application programming interface) for the worker node configured for communication between the master node and the worker node;
• when receiving a task by the master node, requesting the worker node to predict a latency time length according to current CPU utilization and current memory usage of the worker node;
• and returning the predicted latency time length to the master via the API of the worker node;
• and
• after the master node collects predicted latency time lengths of the plurality of worker nodes, assigning the task to a corresponding worker node with a lowest predicted latency time length.