AI Accelerator Virtualization

An AI (Artificial Intelligence) system for Neural Network (NN) Processing, comprising:

• a system processor;
• a system memory device;
• an interconnection interface;
• and
• an AI (Artificial Intelligence) processor coupled to the interconnection interface;
• and
• wherein the AI processor comprises:
• a Core Computing Unit (CCU) comprising at least two Core Computing Elements (CCEs), wherein
• a first Core Computing Element (CCE), corresponding to one level of the CCU, comprises a plurality of Processing Elements (PEs), wherein each PE comprises a multiplier array, an adder tree and an accumulator;
• a second CCE, corresponding to another level of the CCU, coupled to output of the first CCE, wherein the second CCE comprises a plurality of Scalar Elements (SE), and each SE is configured to generate an output of one target activation function for an input to said each SE;
• a unified buffer coupled to the first CCE and the second CCE to store data;
• a control circuitry coupled to the CCU and the unified buffer;
• and
• wherein the AI processor is configured to perform the NN processing for a plurality of users;
• wherein at one time instance:
• at least one of said at least two CCEs is divided into at least two groups to allow at least two users of the plurality of users to share concurrently;
• and
• at least one part of one of said at least two CCEs is allocated to a first user of the plurality of users and at least one part of another of said at least two CCEs is allocated to a second user of the plurality of users, and wherein at another time instance after said one time instance, said at least one part of one of said at least two CCEs is allocated to one next user other than the first user of the plurality of users or said at least one part of another of said at least two CCEs is allocated to one next user other than the second user of the plurality of users.