System and method for input data load adaptive parallel processing

A system comprising a plurality of subsystems, implemented on at least one of hardware logic and software logic executing on a plurality of processors, for computing resource management, the system comprising:

• a first subsystem configured to allocate an array of units of computing capacity among instances of a set of programs, each program of the set of programs comprising a respective plurality of instances, wherein
• the allocating is based at least in part on respective volumes of processing data for each program of the set of programs, and
• the allocating results in a unique subset of the array of units of computing capacity allocated to each program of the set of programs;
• a second subsystem configured to, for each program of the set of programs, assess a relative readiness for execution among the plurality of instances of the respective program, wherein
• the relative readiness is assessed based at least in part on respective volumes of processing data available to each instance of the plurality of instances;
• and
• a third subsystem configured to select, for each program of the set of programs, a highest priority subset of instances of the plurality of instances of the respective program for execution on the array of units of computing capacity according to a number of units of the unique subset of the array of units allocated to the respective program, wherein
• the highest priority subset of instances is selected based at least in part on the relative readiness for execution, and
• selecting results in a plurality of selected instances;
• wherein
• the plurality of selected instances includes
• one or more assigned instances currently executing on a given unit of the array of units of computing capacity, and
• one or more unassigned instances currently unassigned to any unit of the array of units of computing capacity, wherein
• selecting the highest priority subset of instances comprises, for at least one program of the set of programs, prioritizing an unassigned instance of the at least one program over an assigned instance of a given program of the set of programs currently executing on a given unit of the array of units of processing capacity, resulting in preemption of the assigned instance, wherein the prioritizing is based at least in part on newly available data to the at least one program.