Compute cluster with balanced resources

A computer storage memory comprising computer-executable instructions that, when executed by a processor, perform a method of scheduling resources in a computing cluster, the method comprising:

• in a first mode:
• allocating resources to at least one ready job in a ready queue, wherein the at least one ready job has not yet been allocated resources, the resources being allocated to each of the at least one ready job based on a minimum resource requirement specified for the ready job including in order to meet the minimum resource requirement of the at least one ready job, de-allocating an allocated resource from another job allocated in excess of the minimum resource requirement of the another job, and, allocating the de-allocated resource to the at least one ready job;
• in a second mode:
• determining a desired respective allocation of resources for each of the plurality of executing jobs;
• and
• adjusting resource usage by de-allocating a resource from at least one job using more than the respective desired allocation of resources and allocating the de-allocated resource to a job using less than the respective desired allocation of resources, wherein adjusting the resource usage comprises processing executing jobs in inverse order of priority and inverse order of length of time already executing, wherein determining a desired respective allocation of resources for each of the plurality of resources comprises:
• determining a number of resources available for allocation among the plurality of executing jobs;
• determining a weight for each of the plurality of executing jobs;
• computing a value representing the aggregate of the weights of the plurality of executing jobs;
• computing an amount of resources for each of the plurality of executing jobs as a fraction of the resources available for allocation, the fraction being based on the ratio of the weight of the executing job to the value representing the aggregate weights.

A system comprising:

• a plurality of computing nodes providing computing resources, at least one of the computing nodes comprising a processor;
• a scheduler, the scheduler comprising:
• a ready queue holding jobs received from a plurality of clients;
• an allocator, executing on at least one processor, operating in a first mode comprising:
• allocating available computing resources to ready jobs in a ready queue, wherein ready jobs have not yet been allocated resources, the resources being allocated to each of the ready jobs based on a minimum resource requirement specified for the ready job, the available computing resources comprising un-allocated resources and resources allocated and used by executing jobs in excess of respective minimum requirements for the executing jobs,
• the allocator further operating in a second mode comprising:
• determining a desired respective allocation of resources for each of the plurality of executing jobs;
• and
• adjusting resource usage by identifying at least one job using more than the respective desired allocation of resources by evaluating the plurality of executing jobs in reverse order of priority, with lowest priority jobs being evaluated first, de-allocating a resource from the at least one job using more than the respective desired allocation of resources and allocating the de-allocated resource to a job using less than the respective desired allocation of resources, wherein the second mode is initiated in response to an event, the event comprising at least a ready job being assigned resources and an executing job freeing up resources upon termination.