Multi-agent inference

A computer-implemented method comprising:

• launching, by an inference agent in a computing environment, a first instance of a first inference model of a plurality of inference models, the plurality of inference models configured to perform a task, wherein the inference agent is a physical or virtual entity to which a computing resource capacity of the computing environment including a total amount of resources that are available for processing the task is allocated, and wherein the plurality of inference models are generated from trained models using a machine learning algorithm;
• receiving, from a master node and by the inference agent, at least a part of model update information indicating a model update suggestion for the first inference model, wherein the master node is configured to operate as a multi-agent master to perform inference model optimization based on a workload and the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task;
• determining, by the inference agent based on at least the part of model update information, a model update action of upgrade or downgrade of the first inference model to a further inference model of the plurality of inference models;
• and
• performing, by the inference agent, the model update action on the first instance of the first inference model,
• wherein the master node balances quality of service, including accuracy and latency, and computing resource consumption by controlling respective numbers of instances of the plurality of inference models executed in the computing environment,
• wherein the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task comprises at least one of networks, network bandwidth, servers, processing, memory, storage, operating systems, applications, application software licenses, active user accounts and virtual machines,
• wherein a quality of service of the first inference model comprises both an accuracy level and a latency level in processing the workload, and
• wherein the inference agent generates a local action table for the inference models executed thereon by considering both a plurality of local factors and the model update suggestion from the master node, wherein the plurality of local factors include an update algorithm determined by considering a random value, and the inference agent determines actual model update actions to be performed on its executed instances of inference models based on its local action table.

A system comprising:

• one or more processing units;
• and
• a memory coupled to the one or more processing units and storing instructions thereon, the instructions, when executed by the one or more processing units, performing acts comprising:
• determining model update information at least based on a workload related to a task and a computing resource capacity of a computing environment including a total amount of resources that are available for processing the task, the model update information indicating respective model update suggestions for a plurality of inference models configured to perform the task, wherein the plurality of inference models are generated from trained models using a machine learning algorithm;
• and
• distributing the model update information to a plurality of inference agents in the computing environment, the plurality of inference agents having a plurality of instances of the plurality of inference models executed thereon, wherein the inference agents are physical or virtual entities to which the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task are allocated,
• wherein a master node configured to operate as a multi-agent master balances quality of service, including accuracy and latency, and computing resource consumption by controlling respective numbers of instances of the plurality of inference models executed in the computing environment,
• wherein the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task comprises at least one of networks, network bandwidth, servers, processing, memory, storage, operating systems, applications, application software licenses, active user accounts and virtual machines, and
• wherein each of the plurality of inference agents generates a local action table for the inference models executed thereon by considering both a plurality of local factors and the respective model update suggestion from the master node, wherein the plurality of local factors include an update algorithm determined by considering a random value, and each of the plurality of inference agents determines actual model update actions to be performed on their executed instances of inference models based on their local action table.