Workload management with data access awareness in a computing cluster

A system for workload management with data access awareness in a computing cluster, the system comprising:

• a workload manager within the computing cluster, the workload manager including a data requirements evaluator module and a scheduler module;
• and
• a processor executing instructions stored in a memory device, wherein the processor, upon executing the instructions:
• in response to receiving an input workload for scheduling by the workload manager:
• retrieves, by the data requirements evaluator module, a set of inputs from a storage system, wherein the inputs each include:
• an indication of whether the input workload is intensive in Input/Output (I/O) of new data or intensive in I/O of existing data,
• data locality proportions for a set of files associated with the input workload, wherein the data locality proportions specifies a proportion of a total data of the set of files that is respectively stored on each of a plurality of cluster hosts of the computing cluster, and
• data access costs specified for each pair of the plurality of cluster hosts in the computing cluster, wherein the data access costs specifies, for each of the plurality of cluster hosts, a cost of accessing data stored on any other one of the plurality of cluster hosts in the computing cluster, and wherein the data access costs are computed based on networking latencies between the plurality of cluster hosts and storage device access latencies within each of the plurality of cluster hosts;
• generates, by the data requirements evaluator module, a list of the plurality of cluster hosts ranked for performing the input workload according to the set of inputs;
• provides the ranked list of cluster hosts to the scheduler module;
• and
• generates, by the scheduler module, a scheduling of the input workload to certain hosts specified in the ranked list of cluster hosts within the computing cluster where the generated scheduling is optimized with the set of inputs.

A computer program product for workload management with data access awareness in a computing cluster, by a processor, the computer program product embodied on a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

• an executable portion that configures a workload manager within the computing cluster to include a data requirements evaluator module and a scheduler module;
• and
• an executable portion that, in response to receiving an input workload for scheduling by the workload manager:
• retrieves, by the data requirements evaluator module, a set of inputs from a storage system, wherein the inputs each include:
• an indication of whether the input workload is intensive in Input/Output (I/O) of new data or intensive in I/O of existing data,
• data locality proportions for a set of files associated with the input workload, wherein the data locality proportions specifies a proportion of a total data of the set of files that is respectively stored on each of a plurality of cluster hosts of the computing cluster, and
• data access costs specified for each pair of the plurality of cluster hosts in the computing cluster, wherein the data access costs specifies, for each of the plurality of cluster hosts, a cost of accessing data stored on any other one of the plurality of cluster hosts in the computing cluster, and wherein the data access costs are computed based on networking latencies between the plurality of cluster hosts and storage device access latencies within each of the plurality of cluster hosts;
• generates, by the data requirements evaluator module, a list of the plurality of cluster hosts ranked for performing the input workload according to the set of inputs;
• provides the ranked list of cluster hosts to the scheduler module;
• and
• generates, by the scheduler module, a scheduling of the input workload to certain hosts specified in the ranked list of cluster hosts within the computing cluster where the generated scheduling is optimized with the set of inputs.