Deep neural networks (dnn) inference using practical early exit networks

A device, comprising:

• a memory to store data and instructions;
• and
• a processor operable to communicate with the memory, wherein the processor is operable to:
• receive a load forecast for a machine learning model to process received requests and generate inferences for the received requests;
• determine split locations in the machine learning model to divide the machine learning model into a plurality of machine learning model portions based on the load forecast;
• determine a batch size for the requests based on the load forecast;
• receive resource information for available resources for processing the requests;
• select resources of the available resources to execute the plurality of machine learning model portions of the machine learning model;
• and
• output the split locations in the machine learning model, the batch size, and the resources selected.

A method, comprising:

• receiving a load forecast for a machine learning model to process received requests and generate inferences for the received requests;
• receiving resource information for a plurality of available resources for processing the requests;
• trying different combinations of split locations in the machine learning model to divide the machine learning model into a plurality of machine learning model portions;
• determining an estimated runtime for the plurality of machine learning model portions for each of the different combinations of the split locations;
• selecting at least one split location from the different combinations of the split locations to divide the machine learning model into the plurality of machine learning model portions based on the estimated runtime;
• trying different combinations of available resources for executing the machine learning model portions;
• determining an estimated overhead cost for each of the different combinations of available resources;
• and
• selecting resources of the available resources based on minimizing the estimated overhead cost.