Static scheduling and dynamic scheduling for compiler-hinted and self-scheduling multi-engine artificial intelligence (ai) processing unit system

An apparatus, comprising:

• a compiler configured to compile a neural network (NN) model to generate a plurality of operations/threads;
• a memory coupled to the compiler, the memory configured to store the operations/threads of the NN model;
• a thread scheduler coupled to the memory, the thread scheduler configured to schedule the operations/threads of the NN model;
• a multi-engine processing unit that includes a plurality of compute units (CUs);
• a performance monitor configured to monitor runtime performance of the multi-engine processing unit and/or runtime performance of a network coupled to the multi-engine processing unit;
• and
• an executor coupled between the thread scheduler, the multi-engine processing unit and the performance monitor, the executor configured to allocate the operations/threads of the NN model, activate a number of the CUs of the multi-engine processing unit for each of the operations/threads, and change the number of the CUs that are activated based on the runtime performance of the multi-engine processing unit and/or the runtime performance of the network.