Stream-based modular and scalable HW accelerator sub-system with design-time parametric reconfigurable NPU cores

A system, comprising:

• a memory;
• a host processor coupled to the memory;
• a host system bus;
• a plurality of hardware accelerator islands coupled to the host system bus, each including:
• a plurality of processing elements;
• a plurality of streaming engines;
• and
• a stream switch coupled to the plurality of processing elements and to the plurality of streaming engines, wherein the stream switch, in operation, streams data between the plurality of processing elements of the hardware accelerator island, and between the plurality of streaming engines of the hardware accelerator island and the plurality of processing elements of the hardware accelerator island;
• and
• a plurality of unidirectional stream switch connections (SSCONNs) coupled between pairs of stream switches of the plurality of hardware accelerator islands, wherein, the stream switches of the plurality of hardware accelerator islands and the SSCONNs form a run-time reconfigurable interconnection mesh between the plurality of processing elements of the plurality of hardware accelerator islands, and, in operation, the interconnection mesh streams data between processing elements of multiple hardware accelerator islands of the plurality of hardware accelerator islands, wherein at least one of the SSCONNs includes stream link conversion circuitry, which, in operation, coverts data streamed via the at least one SSCONN between stream switches of the hardware accelerator islands operating with different data widths, with different channel configurations, or with different data widths and different channel configurations.

A method, comprising:

• executing a first neural network on a plurality of hardware accelerator islands, the plurality of hardware accelerators being coupled together using a plurality of unidirectional stream switch connections (SSCONNs), each hardware accelerator island including:
• a plurality of processing elements;
• a plurality of streaming engines;
• and
• a stream switch coupled to the plurality of processing elements and to the plurality of streaming engines, wherein the SSCONNs are coupled between pairs of stream switches of the plurality of hardware accelerator islands, and the stream switches of the plurality of hardware accelerator islands and the SSCONNs form a run-time reconfigurable interconnection mesh between the plurality of processing elements of the plurality of hardware accelerator islands, wherein the executing the first neural network includes:
• streaming data between processing elements of a hardware accelerator of the plurality of hardware accelerators using a stream switch of the hardware accelerator of the plurality of hardware accelerators;
• and
• streaming data between processing elements of multiple hardware accelerator islands of the plurality of hardware accelerator islands via the run-time reconfigurable interconnection mesh, wherein streaming data between processing elements of multiple hardware accelerator islands of the plurality of hardware accelerator islands via the run-time reconfigurable interconnection mesh includes converting data streamed via SSCONNs between stream switches of the hardware accelerator islands operating with different data widths, with different channel configurations, or with different data widths and different channel configurations.

A device, comprising:

• memory, which, in operation, stores instructions;
• and
• processing circuitry, coupled to the memory, wherein the processing circuitry, in operation, executes the instructions to program a hardware accelerator system to process an epoch of a neural network, wherein,
• the hardware accelerator system includes:
• a plurality of hardware accelerator islands, each hardware accelerator island including a plurality of processing elements, a plurality of streaming engines, and a stream switch coupled to the plurality of processing elements and to the plurality of streaming engines;
• and
• a plurality of unidirectional stream switch connections (SSCONNs) coupled between pairs of stream switches of the plurality of hardware accelerator islands, wherein, the stream switches of the plurality of hardware accelerator islands and the SSCONNs form a run-time reconfigurable interconnection mesh between the plurality of processing elements of the plurality of hardware accelerator islands, and
• the programming the hardware accelerator system to process an epoch of a neural network includes:
• programming a stream switch of an island to stream an intra island data stream associated with the epoch;
• and
• programming the reconfigurable interconnection mesh to stream an inter island data stream associated with the epoch, wherein at least one of the SSCONNs includes stream link conversion circuitry, and the programming the reconfigurable interconnection mesh to stream the inter island data stream associated with the epoch includes programming the stream link conversion circuitry to covert data streamed via the at least one SSCONN between stream switches of the hardware accelerator islands operating with different data widths, with different channel configurations, or with different data widths and different channel configurations.