Optimizing inference for deep-learning neural networks in a heterogeneous system

A method for generating an artificial neural network (ANN), the method comprising:

• inputting specifications of a target inference device to an ANN generation device;
• generating candidate ANNs, by the ANN generation device, based on the specifications;
• generating trained ANNs, by the ANN generation device, by training the candidate ANNs to perform an inference task;
• generating profiles of the trained ANNs, wherein the profiles indicate characteristics of the trained ANNs;
• storing the profiles to be queried based on the requirements and to return a profile of one of the trained ANNs having characteristics satisfying requirements of an application, for deployment on a target inference device.

A device for generating an artificial neural network (ANN), the device comprising:

• an input interface configured to input specifications of a target inference device;
• processing circuitry configured to generate candidate ANNs based on the specifications;
• processing circuitry configured to generate trained ANNs by training the candidate ANNs to perform an inference task;
• profiling circuitry configured to generate profiles of the trained ANNs which reflect the characteristics of each trained ANN, and to store the profiles to be queried based on the requirements and to return a profile of one of the trained ANNs having characteristics satisfying requirements of an application, for deployment on a target inference device.

A device for deploying an artificial neural network (ANN) to perform an inference task, the device comprising:

• an input interface configured to input inference task requirements of an application;
• querying circuitry configured to query stored profiles based on the requirements, the profiles reflecting characteristics of ANNs trained to perform an inference task on a target inference device;
• the querying circuitry further configured to select an ANN based on the query;
• and
• deployment circuitry configured to deploy the selected ANN on an inference device conforming to specifications of the target inference device.