Heterogeneous compute-based artificial intelligence model partitioning

At least one non-transitory machine-readable medium including instructions for heterogeneous compute-based artificial intelligence model partitioning that, when executed by at least one processor, cause the at least one processor to perform operations to:

• generate an intermediate representation of an input machine learning model by performing topological sorting of nodes of the input machine learning model to generate a directed acyclic graph (DAG) intermediate representation;
• analyze the intermediate representation to determine compute metrics for execution of the input machine learning model;
• analyze an input processing device to determine normalization metrics for execution of the input machine learning model on the input processing device;
• generate a partition of the intermediate representation for the input processing device based on the compute metrics and the normalization metrics, wherein the partition is compiled using a graph compiler to emit a device-specific executable optimized for the input processing device;
• automatically deploy the compiled partition to the input processing device for execution, thereby reducing model load-latency and improving edge inference performance;
• and
• transmit the partition to the input processing device for execution.

A computer-implemented method for allocating a workload to hardware in a heterogeneous computing environment, comprising:

• executing, by at least one processor, a machine learning model to analyze one or more characteristics of a workload and one or more attributes of a plurality of available hardware devices;
• determining, by the at least one processor and based on analysis by the machine learning model, one or more recommended hardware devices from the plurality of available hardware devices for execution of the workload;
• performing topological sorting of nodes of the workload or the machine learning model to generate a directed acyclic graph (DAG) intermediate representation;
• partitioning, by the at least one processor, the DAG intermediate representation into a plurality of sub-tasks;
• compiling, by the at least one processor, each partition using a graph compiler to emit a device-specific executable optimized for a respective recommended hardware device;
• automatically distributing, by the at least one processor, the plurality of sub-tasks amongst the one or more recommended hardware devices based on device-specific compute metrics of the one or more recommended hardware devices using respective device-specific executables;
• and
• causing, by the at least one processor, execution of the plurality of sub-tasks on a respective recommended hardware device of the one or more recommended hardware devices.