US 2024/0177024 A1 · SYSTEM AND METHOD FOR MANAGING INFERENCE MODELS BASED ON INFERENCE GENERATION FREQUENCIES

Abstract

Methods and systems for managing execution of an inference model hosted by data processing systems are disclosed. To manage execution of the inference model, a system may include an inference model manager and any number of data processing systems. The inference model manager may identify an inference frequency capability of the inference model hosted by the data processing systems and may determine whether the inference frequency capability of the inference model meets an inference frequency requirement of a downstream consumer during a future period of time. If the inference frequency capability does not meet the inference frequency requirement of the downstream consumer, the inference model manager may modify a deployment of the first inference model to meet the inference frequency requirement of the downstream consumer.

Claims20

Dependent claims

A method of managing execution of a first inference model hosted by data processing systems, the method comprising:

obtaining an inference frequency capability of the first inference model, the inference frequency capability indicating a rate of execution of the first inference model;
making a first determination regarding whether the inference frequency capability of the first inference model meets an inference frequency requirement of a downstream consumer during a future period of time;
in an instance of the first determination in which the inference frequency capability of the first inference model does not meet the inference frequency requirement of the downstream consumer:
obtaining an execution plan for the first inference model based on the inference frequency requirement of the downstream consumer;
and
prior to the future period of time, modifying a deployment of the first inference model to the data processing systems based on the execution plan.

3 more elements

A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations for managing execution of a first inference model hosted by data processing systems, the operations comprising:

obtaining an inference frequency capability of the first inference model, the inference frequency capability indicating a rate of execution of the first inference model;
making a first determination regarding whether the inference frequency capability of the first inference model meets an inference frequency requirement of a downstream consumer during a future period of time;
in an instance of the first determination in which the inference frequency capability of the first inference model does not meet the inference frequency requirement of the downstream consumer:
obtaining an execution plan for the first inference model based on the inference frequency requirement of the downstream consumer;
and
prior to the future period of time, modifying a deployment of the first inference model to the data processing systems based on the execution plan.

3 more elements

A data processing system, comprising:

a processor;
and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations for managing execution of a first inference model hosted by data processing systems, the operations comprising:
obtaining an inference frequency capability of the first inference model, the inference frequency capability indicating a rate of execution of the first inference model;
making a first determination regarding whether the inference frequency capability of the first inference model meets an inference frequency requirement of a downstream consumer during a future period of time;
in an instance of the first determination in which the inference frequency capability of the first inference model does not meet the inference frequency requirement of the downstream consumer:
obtaining an execution plan for the first inference model based on the inference frequency requirement of the downstream consumer;
and
prior to the future period of time, modifying a deployment of the first inference model to the data processing systems based on the execution plan.

6 more elements

Dependencies are inferred from the claim text.

System and method for managing inference models based on inference generation frequencies

Life of this patent

System and method for managing inference models based on inference generation frequencies

Life of this patent

Abstract

Claims20