Adaptive dispatch for acceleration of deep neural networks on graphic processing units

An apparatus for processing at a device, comprising:

• a processor, memory coupled with the processor;
• and instructions stored in the memory and executable by the processor to cause the apparatus to:
• parse a set of layers of a deep neural network, wherein the set of layers is associated with a set of machine learning operations of the deep neural network;
• determine one or more layer parameters based at least in part on the determined set of layers;
• determine an execution time associated with executing a shader dispatch based at least in part on the one or more layer parameters;
• insert a flush marker into a shader sequence based at least in part on determining a target execution time and the determined execution time associated with the shader dispatch;
• batch the shader dispatch to a command buffer based at least in part on the execution time, wherein the shader dispatch is batched to the command buffer based at least in part on the flush marker;
• and
• process the command buffer based at least in part on the batching.

A method for processing at a device, comprising:

• parsing a set of layers of a deep neural network, wherein the set of layers is associated with a set of machine learning operations of the deep neural network;
• identifying a first layer of the set of layers based at least in part on the parsing;
• determining a shader sequence based at least in part on the identified first layer;
• determining one or more layer parameters based at least in part on the determined set of layers;
• determining an execution time associated with executing a shader dispatch based at least in part on the one or more layer parameters;
• dispatching the shader sequence into a set of shader dispatches, wherein the set of shader dispatches includes the shader dispatch;
• batching the shader dispatch to a command buffer based at least in part on the execution time;
• and
• processing the command buffer based at least in part on the batching.