Hardware adaptive multi-model scheduling

A computer-implemented method for multi-model scheduling, comprising:

• given a performance metric array for a group of neural network models to provide an artificial intelligence (AI) based service in a heterogeneous hardware platform comprising multiple edge devices, identifying a worst performance parameter in the performance metric array and an edge device associated with the worst performance parameter;
• identifying a neural network model, among the group of neural network models, with the best performance parameter for the identified edge device;
• allocating the identified neural network model to the identified edge device;
• and
• iteratively updating the performance metric array with the allocated edge device and the allocated neural network removed for subsequent identifying and allocation processes, until the group of neural network models are allocated to obtain a first set of allocation results.

A non-transitory computer-readable medium or media, comprising one or more sequences of instructions which, when executed by at least one processor, causes steps for multi-model scheduling comprising:

• obtaining a performance metric array for a group of neural network models to provide an artificial intelligence (AI) based service in a heterogeneous hardware platform comprising multiple edge devices, the performance metric array comprises a plurality of performance parameters, each performance parameter is obtained by independently running one neural network model, among the group of neural network models, on one of the multiple edge devices;
• implementing a model first scheduling (MFS) process and a hardware first scheduling (HFS) from the performance metric array to obtain MFS allocation results and HSF allocation results respectively, the MFS allocation results define allocations between the group of neural network models and the multiple edge devices using the MFS process, the HFS allocation results define allocations between the group of neural network models and the multiple edge devices using the HFS process;
• and
• responsive to the MFS allocation results being different from the HFS allocation results, running the group of neural network models on the heterogeneous hardware platform based on the MFS allocation results and the HFS allocation results respectively for a performance comparison, the allocation results, between the MFS allocation results and the HFS allocation results, having higher performance are chosen for model allocations.