Hardware and software co-designed system for efficient distributed control of execution on a compute accelerator

An AI accelerator apparatus, the apparatus comprising:

• a plurality of chiplets, each of the chiplets comprising a plurality of tiles, and each of the tiles comprising a plurality of slices, a central processing unit (CPU) coupled to the plurality of slices, and a dispatch engine device coupled to the CPU;
• wherein the dispatch engine device comprises:
• a task queue module having a plurality of cells, the task queue module being configured to receive one or more task groups, each of the task groups having a plurality of work units configured in a hierarchy of queues, and wherein the task queue module is configured to store each of the plurality of work units of each task group in one of the plurality of cells such that the plurality of work units of each task group are stored according to the hierarchy of queues;
• a bus device coupled between the plurality of cells in the task queue module and the plurality of slices such that each cell is configured to communicate through the bus device to one of the plurality of slices;
• a completion handler module coupled between the plurality of slices and each of the plurality of cells in the task queue module and configured to send a completion signal upon a completion condition to update a status of one or more of the plurality of cells;
• a resource monitor module coupled to each of the plurality of cells in the task queue module and configured to monitor a state of each of the slices;
• and
• a task dispatcher module coupled to each of the plurality of cells in the task queue module and configured to asynchronously dispatch the work unit stored in each of the plurality of cells to one of the plurality of slices in a parallel pipeline process using the bus device;
• a plurality of die-to-die (D2D) interconnects coupled to the each of CPUs in each of the tiles;
• a peripheral component interconnect express (PCIe) bus coupled to the CPUs in each of the tiles;
• a dynamic random access memory (DRAM) interface coupled to the CPUs in each of the tiles;
• a global CPU coupled to each of the CPUs in each of the tiles;
• and
• a global dispatch engine coupled to the global CPU and configured to cause the global CPU to transfer the one or more task groups to each CPU in the plurality of chiplet devices, and wherein each CPU is configured to transfer each of the plurality of work units of the one or more task groups into one of the plurality of cells in the task queue module according to the hierarchy of queues.