System and method for executing multiple inference models using inference model prioritization

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing inference models hosted by data processing systems to complete timely execution of the inference models, the operations comprising:

• making a first determination, based on a result of a compliance analysis of operational capability data obtained from the data processing systems and a result of a capacity analysis of the operational capability data, regarding whether the inference models are likely to complete timely execution;
• in a first instance of the first determination where the inference models are unlikely to complete timely execution:
• making a second determination, based on the result of the capacity analysis, regarding whether the data processing systems have capacity to host a total quantity of inference models specified by an execution plan;
• in a first instance of the second determination where the data processing systems have insufficient capacity to host the total quantity of inference models:
• modifying the execution plan to obtain an updated execution plan, the execution plan being modified to retain a first assurance level for a first type of inference model of the inference models and reducing a second assurance level for a second type of inference model of the inference models, and
• modifying a deployment of the inference models based on the updated execution plan.

A data processing system, comprising:

• a processor;
• and
• a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing inference models hosted by data processing systems to complete timely execution of the inference models, the operations comprising:
• making a first determination, based on a result of a compliance analysis of operational capability data obtained from the data processing systems and a result of a capacity analysis of the operational capability data, regarding whether the inference models are likely to complete timely execution;
• in a first instance of the first determination where the inference models are unlikely to complete timely execution:
• making a second determination, based on the result of the capacity analysis, regarding whether the data processing systems have capacity to host a total quantity of inference models specified by an execution plan;
• in a first instance of the second determination where the data processing systems have insufficient capacity to host the total quantity of inference models:
• modifying the execution plan to obtain an updated execution plan, the execution plan being modified to retain a first assurance level for a first type of inference model of the inference models and reducing a second assurance level for a second type of inference model of the inference models, and
• modifying a deployment of the inference models based on the updated execution plan.