System and method for inference generation through dynamic reassignment of inference model portions

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for executing an inference model across multiple data processing systems that each individually have insufficient computing resources to complete timely execution of the inference model, the operations comprising:

• initiating execution of an inference model using portions of the inference model distributed to the data processing systems to obtain an inference model result, at least two of the portions of the inference model generating redundant processing results as specified by an execution plan for the inference model to facilitate continued execution of the inference model in an event of termination of a host of one of the at least two of the portions of the inference model;
• and
• while the inference model is executing and at least one of the data processing systems becomes unable to complete execution of one of the distributed portions of the inference model:
• managing the execution of the inference model based on the execution plan and through, at least in part, distribution of additional copies of the portions of the inference model to the data processing systems to provide eventual compliance with the execution plan and through which the inference model result is timely obtained, the timely obtained inference model result being relied upon by a downstream consumer for operation.

A data processing system, comprising:

• a processor;
• and
• a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for executing an inference model across multiple data processing systems that each individually have insufficient computing resources to complete timely execution of the inference model, the operations comprising:
• initiating execution of an inference model using portions of the inference model distributed to the data processing systems to obtain an inference model result, at least two of the portions of the inference model generating redundant processing results as specified by an execution plan for the inference model to facilitate continued execution of the inference model in an event of termination of a host of one of the at least two of the portions of the inference model;
• and
• while the inference model is executing and at least one of the data processing systems becomes unable to complete execution of one of the distributed portions of the inference model;
• managing the execution of the inference model based on the execution plan and through, at least in part, distribution of additional copies of the portions of the inference model to the data processing systems to provide eventual compliance with the execution plan and through which the inference model result is timely obtained, the timely obtained inference model result being relied upon by a downstream consumer for operation.