Optimized assignments and/or generation virtual machine for reducer tasks

A system for determining and executing optimal virtual machine assignments for reducer tasks on physical hosts comprising:

• at least one memory element;
• at least one processor coupled to the at least one memory element;
• a costs module that when executed by the at least one processor is configured to:
• determine a distribution of keys over mapper virtual machines after map tasks are complete;
• and
• determine costs associated with assignments of virtual machines to reducer tasks on the keys based on the distribution of keys;
• defining a variable matrix for assigning virtual machines to reducer tasks, wherein the variable matrix comprises values indicating whether a virtual machine is to be assigned to reduce one or more keys, and the variable matrix has dimensions of at least n by (M+p*q), where n is the number of keys, M is the number of mapper virtual machines, p is n-M, and q is the number of available physical hosts on which a virtual machine can be created;
• and
• a constraints solver that when executed by the at least one processor is configured to:
• determine the optimal virtual machine assignments for the reducer tasks based on the variable matrix and the costs, subject to one or more constraints on the variable matrix;
• and
• providing the optimal virtual machine assignments for the reducer tasks to a scheduler to execute the optimal virtual machine assignments on the physical hosts.

A computer-readable non-transitory medium comprising one or more instructions, for determining and executing optimal virtual machine assignments for reducer tasks on physical hosts, that when executed on a processor configure the processor to perform one or more operations comprising:

• determining a distribution of keys over mapper virtual machines after map tasks are complete;
• determining costs associated with assignments of virtual machines to reducer tasks on the keys based on the distribution of keys;
• defining a variable matrix for assigning virtual machines to reducer tasks, wherein the variable matrix comprises values indicating whether a virtual machine is to be assigned to reduce one or more keys, and the variable matrix has dimensions of at least n by (M+p*q), where n is the number of keys, M is the number of mapper virtual machines, p is n-M, and q is the number of available physical hosts on which a virtual machine can be created;
• determining the optimal virtual machine assignments for the reducer tasks, using a constraints solver, based on the variable matrix and the costs, subject to one or more constraints on the variable matrix;
• and
• assigning, according to the optimal virtual machine assignments, the reducer tasks for execution by the virtual machines on the physical hosts.