# US 12,602,603 B2 · Multi-agent inference > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,603 B2 - Document ID: US12602603B2 - Status: Granted - Kind: B2 - Application number: 17/330,099 - Filed: 2021-05-25 - Published: 2022-12-01 - Granted: 2026-04-14 - Earliest claimed: 2021-05-25 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Li Cao; Yi Shan Jiang; Ze Ming Zhao; Hong Bo Peng - Web page: https://patentgrep.com/patent/US12602603B2 ## CPC - G06N 5/04: Inference or reasoning models - G06N 5/043: Distributed expert systems; Blackboards - G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks - G06N 20/00: Machine learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A computer-implemented method includes determining, by a master node, model update information at least based on a workload related to a task and a resource capacity of a computing environment. The model update information indicates respective model update suggestions for a plurality of inference models configured to perform the task. The method further includes distributing, by the master node, the model update information to a plurality of inference agents in the computing environment. The plurality of inference agents has a plurality of instances of the plurality of inference models executed thereon. ## Claims ### Claim 1 A computer-implemented method comprising: determining, by a master node, model update information at least based on a workload related to a task and a computing resource capacity of a computing environment including a total amount of resources that are available for processing the task, the model update information indicating respective model update suggestions for a plurality of inference models configured to perform the task, the master node configured to operate as a multi-agent master to perform inference model optimization based on the workload and the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task, wherein the plurality of inference models are generated from trained models using a machine learning algorithm; and distributing, by the master node, the model update information to a plurality of inference agents in the computing environment, the inference agents being physical or virtual entities to which the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task are allocated, the plurality of inference agents having a plurality of instances of the plurality of inference models executed thereon, wherein the master node balances quality of service, including accuracy and latency, and computing resource consumption by controlling respective numbers of instances of the plurality of inference models executed in the computing environment, wherein the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task comprises at least one of networks, network bandwidth, servers, processing, memory, storage, operating systems, applications, application software licenses, active user accounts and virtual machines, and wherein each of the plurality of inference agents generates a local action table for the inference models executed thereon by considering both a plurality of local factors and the respective model update suggestion from the master node, wherein the plurality of local factors include an update algorithm determined by considering a random value, and each of the plurality of inference agents determines actual model update actions to be performed on their executed instances of inference models based on their local action table. ### Claim 2 (depends on 1) The method of claim 1, wherein the respective model update suggestions comprise at least one of the following: a first probability of upgrading a first inference model of the plurality of inference models to a second inference model of the plurality of inference models, the first inference model having at least one of a lower quality of service or a lower computing resource consumption level than the second inference model, and a second probability of downgrading the first inference model to a third inference model of the plurality of inference models, the first inference model having at least one of a higher quality of service or a higher computing resource consumption level than the third inference model. ### Claim 3 (depends on 2) The method of claim 2, wherein a quality of service of the first inference model comprises both an accuracy level and a latency level in processing the workload. ### Claim 4 (depends on 1) The method of claim 1, wherein distributing the model update information comprises: in accordance with a determination that a first inference agent of the plurality of inference agent has an instance of a first inference model of the plurality of inference models, providing a part of the model update information related to the first inference model to the first inference agent. ### Claim 5 (depends on 1) The method of claim 1, wherein determining the model update information comprises: determining the model update information further based on at least one of the following: respective qualities of service of the plurality of inference models, and respective computing resource consumption levels of the plurality of inference models. ### Claim 6 (depends on 1) The method of claim 1, wherein determining the model update information comprises: determining the model update information by applying an exploit-explore trade-off policy. ### Claim 7 (depends on 6) The method of claim 6, wherein determining the model update information by applying the exploit-explore trade-off policy comprises: determining respective candidate model update suggestions for the plurality of inference models; determining a reward of the respective candidate model update suggestions, the reward indicating an expected improvement level to a predetermined objective in a condition of the workload and the computing resource capacity including a total amount of resources that are available for processing the task; and determining the model update information based on the reward. ### Claim 8 (depends on 1) The method of claim 1, wherein the model update information is determined and distributed periodically. ### Claim 9 A computer-implemented method comprising: launching, by an inference agent in a computing environment, a first instance of a first inference model of a plurality of inference models, the plurality of inference models configured to perform a task, wherein the inference agent is a physical or virtual entity to which a computing resource capacity of the computing environment including a total amount of resources that are available for processing the task is allocated, and wherein the plurality of inference models are generated from trained models using a machine learning algorithm; receiving, from a master node and by the inference agent, at least a part of model update information indicating a model update suggestion for the first inference model, wherein the master node is configured to operate as a multi-agent master to perform inference model optimization based on a workload and the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task; determining, by the inference agent based on at least the part of model update information, a model update action of upgrade or downgrade of the first inference model to a further inference model of the plurality of inference models; and performing, by the inference agent, the model update action on the first instance of the first inference model, wherein the master node balances quality of service, including accuracy and latency, and computing resource consumption by controlling respective numbers of instances of the plurality of inference models executed in the computing environment, wherein the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task comprises at least one of networks, network bandwidth, servers, processing, memory, storage, operating systems, applications, application software licenses, active user accounts and virtual machines, wherein a quality of service of the first inference model comprises both an accuracy level and a latency level in processing the workload, and wherein the inference agent generates a local action table for the inference models executed thereon by considering both a plurality of local factors and the model update suggestion from the master node, wherein the plurality of local factors include an update algorithm determined by considering a random value, and the inference agent determines actual model update actions to be performed on its executed instances of inference models based on its local action table. ### Claim 10 (depends on 9) The method of claim 9, wherein the model update information comprises at least one of: respective qualities of service of the plurality of inference models, and respective computing resource consumption levels of the plurality of inference models. ### Claim 11 (depends on 9) The method of claim 9, wherein the model update suggestion comprises at least one of the following: a first probability of upgrading the first inference model to a second inference model of the plurality of inference models, the first inference model having a lower quality of service and/or a lower computing resource consumption level than the second inference model, and a second probability of downgrading the first inference model to a third inference model of the plurality of inference models, the first inference model having a higher quality of service and/or a higher computing resource consumption level than the third inference model. ### Claim 12 (depends on 9) The method of claim 9, wherein performing the model update action comprises: in accordance with a determination that the model update action indicates the upgrade or the downgrade, launching a second instance of the further inference model on the inference agent. ### Claim 13 A system comprising: one or more processing units; and a memory coupled to the one or more processing units and storing instructions thereon, the instructions, when executed by the one or more processing units, performing acts comprising: determining model update information at least based on a workload related to a task and a computing resource capacity of a computing environment including a total amount of resources that are available for processing the task, the model update information indicating respective model update suggestions for a plurality of inference models configured to perform the task, wherein the plurality of inference models are generated from trained models using a machine learning algorithm; and distributing the model update information to a plurality of inference agents in the computing environment, the plurality of inference agents having a plurality of instances of the plurality of inference models executed thereon, wherein the inference agents are physical or virtual entities to which the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task are allocated, wherein a master node configured to operate as a multi-agent master balances quality of service, including accuracy and latency, and computing resource consumption by controlling respective numbers of instances of the plurality of inference models executed in the computing environment, wherein the computing resource capacity of the computing environment including a total amount of resources that are available for processing the task comprises at least one of networks, network bandwidth, servers, processing, memory, storage, operating systems, applications, application software licenses, active user accounts and virtual machines, and wherein each of the plurality of inference agents generates a local action table for the inference models executed thereon by considering both a plurality of local factors and the respective model update suggestion from the master node, wherein the plurality of local factors include an update algorithm determined by considering a random value, and each of the plurality of inference agents determines actual model update actions to be performed on their executed instances of inference models based on their local action table. ### Claim 14 (depends on 13) The system of claim 13, wherein the respective model update suggestions comprise at least one of the following: a first probability of upgrading a first inference model of the plurality of inference models to a second inference model of the plurality of inference models, the first inference model having a lower quality of service and/or a lower computing resource consumption level than the second inference model, and a second probability of downgrading the first inference model to a third inference model of the plurality of inference models, the first inference model having a higher quality of service and/or a higher computing resource consumption level than the third inference model. ### Claim 15 (depends on 14) The system of claim 14, wherein a quality of service of the first inference model comprises both an accuracy level and a latency level in processing the workload. ### Claim 16 (depends on 13) The system of claim 13, wherein distributing the model update information comprises: in accordance with a determination that a first inference agent of the plurality of inference agent has an instance of a first inference model of the plurality of inference models, providing a part of the model update information related to the first inference model to the first inference agent. ### Claim 17 (depends on 13) The system of claim 13, wherein determining the model update information comprises: determining the model update information further based on at least one of the following: respective qualities of service of the plurality of inference models, and respective computing resource consumption levels of the plurality of inference models. ### Claim 18 (depends on 13) The system of claim 13, wherein determining the model update information comprises: determining the model update information by applying an exploit-explore trade-off policy. ### Claim 19 (depends on 18) The system of claim 18, wherein determining the model update information by applying the exploit-explore trade-off policy comprises: determining respective candidate model update suggestions for the plurality of inference models; determining a reward of the respective candidate model update suggestions, the reward indicating an expected improvement level to a predetermined objective in a condition of the workload and the computing resource capacity including a total amount of resources that are available for processing the task; and determining the model update information based on the reward. ### Claim 20 (depends on 13) The system of claim 13, wherein the model update information is determined and distributed periodically.