Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system

A Hadoop computer system comprising:

• a first Hadoop node comprising one or more processors and configured to perform computing processes on tasks assigned to the first Hadoop node;
• a second Hadoop node comprising one or more processors and configured to perform computing processes on tasks assigned to the second Hadoop node;
• a resource controller configured to receive a computing job submission, wherein the resource controller is further configured to divide the received computing job submission into a first task and a second task, and to assign the first task to the first Hadoop node and to assign the second task to the second Hadoop node;
• the first Hadoop node comprising a first node controller, the first node controller configured to transmit first node memory information that includes data indicating memory resource availability of the first Hadoop node;
• and
• the second Hadoop node comprising a second node controller, the second node controller configured to transmit second node memory information that includes data indicating memory resource availability of the second Hadoop node,
• wherein the resource controller is configured to receive the first node memory information from the first Hadoop node and, based at least in part on the data indicating memory resource availability of the first Hadoop node, to determine whether to increase memory for use by tasks of the first Hadoop node, and whether to decrease memory for use by tasks of the first Hadoop node;
• and
• wherein the resource controller is configured to receive the second node memory information from the second Hadoop node and, based at least in part on the data indicating memory resource availability of the second Hadoop node, to determine whether to increase memory for use by tasks of the second Hadoop node, and whether to decrease memory for use by tasks of the second Hadoop node.