Preserving quality of service for client applications having workloads for execution by a compute core or a hardware accelerator

A system comprising:

• an accelerator portal to allow a plurality of client applications access to one or more of a plurality of shared hardware accelerators, wherein each of the plurality of client applications can execute a workload using a compute core or by using one of the plurality of shared hardware accelerators;
• a hardware accelerator, from among the plurality of shared hardware accelerators, configured to queue commands for execution of workloads in a first set of command queues;
• the hardware accelerator further configured to queue commands for execution of workloads in a second set of command queues;
• and
• workload processors, associated with the hardware accelerator, configured to execute workloads specified by the commands in the first set of command queues and the second set of command queues in an order of execution that is determined based on output of a set of trackers configured to track one or more criteria for a selected set of command queues from among the first set of command queues and the second set of command queues.

A method for operating a hardware accelerator configured to process commands submitted by client applications directly to the hardware accelerator, wherein a workload corresponding to a command can be executed either by a compute core accessible to a client application or by the hardware accelerator, the method comprising:

• queueing commands for execution of workloads in a first set of command queues, wherein each of the first set of command queues is associated with one or more of a set of virtual functions for a physical function associated with the hardware accelerator, and wherein a first set of trackers, configured to track one or more a first set of criteria for a first selected set of command queues from among the first set of command queues, is assigned by a virtual machine for a first stage of arbitration;
• queueing commands for execution of workloads in a second set of command queues, wherein a first subset of the second set of command queues is associated with the set of virtual functions for the physical function, wherein a second set of trackers, configured to track one or more of a second set of criteria for a second selected set of command queues from among the second set of command queues, is assigned by a system agent for a second stage of arbitration;
• and
• workload processors associated with the hardware accelerator executing workloads specified by commands in the first set of command queues and the second set of command queues in an order of execution that is determined based on tracking by a subset of the first set of trackers and the second set of trackers.