Neural network scheduling mechanism

A method to facilitate workload scheduling, comprising:

• receiving requests from one or more clients to access hardware resources of a general purpose graphics processing unit;
• scheduling direct access of the hardware resources to a first client of the one or more clients to process the workloads, wherein the one or more clients are each associated with a precompiled neural network (NN) kernel;
• and
• gathering, via a gather unit of the general purpose graphics processing unit, non-zero data values associated with the one or more clients while bypassing zero data values associated with the one or more clients.

At least one non-transitory computer readable medium having instructions, which when executed by one or more processors, the one or more processors including a general purpose graphics processing unit, cause the one or more processors to:

• receive requests from one or more clients to access hardware resources of the general purpose graphics processing unit to process workloads associated with the one or more clients;
• schedule direct access of the hardware resources to a first client of the one or more clients to process the workloads, wherein the one or more clients are each associated with a precompiled neural network (NN) kernel;
• and
• gathering, via a gather unit of the general purpose graphics processing unit, non-zero data values associated with the one or more clients while bypassing zero data values associated with the one or more clients.