System and method of scheduling and computing resource allocation optimization of machine learning flows

A system for computing resource allocation optimization for machine learning flows based on established hardware constraints in a single computing unit, comprising:

• A computing unit comprising:
• at least one processor, wherein said at least one processor comprises processor constraints;
• at least one memory module, wherein said at least one memory module comprises memory constraints;
• at least one block device, wherein said at least one block device comprises block device constraints;
• an execution matrix, wherein said execution matrix represents a process flow;
• an optimizing module configured to:
• establish a maximum number of optimization iterations;
• establish a hardware performance cost limit, wherein said hardware performance cost limit comprises a memory cost limit, a processor cost limit and a block device cost limit;
• receive data size estimates based on data to be processed by said process flow;
• determine an execution pattern, wherein said execution pattern comprises a plurality of operators that form said process flow;
• assign hardware resources from said computing unit to the process flow represented by said execution matrix
• determine a plurality of resource allocation configurations by performing the steps of:
• a) determining processor bursts and minimizing overlap within processor cores based on said processor constraints, said execution pattern, and said data size estimates, by assigning processor cores to said process flow that do not exceed said processor cost limit;
• b) determining a memory pattern layout to minimize cache misses and pre-fetching out of order based on said memory constraints, said execution pattern, and said data size estimates, by assigning memory access to said process flow without exceeding said memory cost limit;
• c) determining a block device pattern layout to minimize hard disk passes and fragmentation based on said block device constraints, said execution pattern, and said data size estimates, by assigning block device access to said process flow without exceeding said block device cost limit;
• d) iterating steps a through c for said maximum number of optimization iterations;
• and
• determine the resource allocation configuration with the lowest hardware performance cost from said plurality of resource allocation configurations.