Coordination and increased utilization of graphics processors during inference

A system comprising:

• a memory device;
• and
• one or more processors including one or more graphics processors, the one or more processors to execute instruction stored on the memory device, wherein the instructions cause the one or more processors to perform operations comprising:
• performing processing for one or more contexts of a client of the system, the system including a plurality of clients;
• and
• accessing scheduled resources of the system for the one or more contexts, the resources being shared by the plurality of clients, wherein shared resources of the system include the one or more graphics processors;
• wherein access to the one or more graphics processors by the one or more contexts is subject to a limitation on usage set by the system;
• and
• wherein the limitation on usage includes a limitation on threads of the one or more graphics processors for the one or more contexts to a specified portion of available threads, the specified portion being less than all available threads of the one or more graphics processors.

An apparatus comprising:

• a processing system including one or more processors including one or more processors, the one or more processors including one or more graphics processors;
• a scheduler to schedule shared resources in the system for a plurality of contexts of clients of the processing system;
• and
• wherein the processing system has a capability to limit usage of processing resources of the one or more graphics processors by the plurality of contexts, wherein the limitation on usage of the one or more graphics processors includes to limit threads for one or more contexts of the plurality of contexts to a specified portion of available threads of the one or more graphics processors, the specified portion being less than all available threads of the one or more graphics processors.