Dynamically reallocating resources for optimized job performance in distributed heterogeneous computer system

A distributed heterogeneous computer system, comprising:

• a plurality of computer nodes, each of the plurality of computer nodes being operatively connected through a network interface to a network to provide data communication between the plurality of computer nodes,
• wherein each of the plurality of computer nodes includes:
• an application layer;
• an optimization layer configured to be decoupled from the application layer;
• and
• one or more processors configured to perform operations of the application layer and the optimization layer,
• wherein the application layer includes:
• an application injector configured to invoke at least one job to be performed;
• and
• an application specific component configured to share a task type of the at least one job, which indicates whether the at least one job is normal or critical, and a data load for the at least one job to the application injector,
• wherein the optimization layer includes:
• a collector configured to dynamically collect a workload value being a function of at least one of a performance metric, a reliability metric, an extra factor, and a data distribution ratio on the each of the plurality of computer nodes;
• a job informer configured to determine respective types of known and unknown jobs on the each of the plurality of computer nodes;
• a job optimizer configured to determine a data distribution vector based on the workload value and the respective types of the known and unknown jobs on each of the computer nodes, the data distribution vector defining a respective amount of the data load to be distribute among the plurality of computer nodes to perform the at least one job;
• a lookup table configured to store the data distribution vector;
• and
• a lookup table manager configured to maintain the lookup table and provide a query interface to the application injector,
• wherein the application injector is further configured to send a query for a task identifier of the at least one job to the lookup table manager through the query interface, receive the data distribution vector from the lookup table manger through the query interface, and schedule the at least one job based on the data distribution vector,
• wherein the lookup table manager is further configured to transmit the data distribution vector from the lookup table to the application injector in response to the query,
• wherein the application injector is further configured to return a time-variance taken per byte to the lookup table manager after scheduling the at least one job based on the data distribution vector,
• wherein the lookup table manager is further configured to update a variance entry in the lookup table when the returned time-variance is greater than a current value of the variance entry in the lookup table.