# US 12,192,292 B2 · Multiple model injection for a deployment cluster > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,192,292 B2 - Document ID: US12192292B2 - Status: Granted - Kind: B2 - Application number: 18/610,561 - Filed: 2024-03-20 - Published: 2024-07-04 - Granted: 2025-01-07 - Earliest claimed: 2020-03-04 - Assignees: Hewlett Packard Enterprise Development LP - Applicants: Hewlett Packard Enterprise Development LP - Inventors: Kartik Mathur - Web page: https://patentgrep.com/patent/US12192292B2 ## CPC - H04L 67/34: {involving the movement of software or configuration parameters (network booting or remote initial program loading [RIPL] G06F9/4416)} - G06F 8/36: Software reuse - G06F 9/44505: {Configuring for program initiating, e.g. using registry, configuration files} - G06F 9/44536: {Selecting among different versions} - G06F 9/45558: {Hypervisor-specific management and integration aspects} - G06F 9/5077: {Logical partitioning of resources; Management or configuration of virtualized resources (specific details on emulation or internal functioning of virtual machines G06F9/455)} - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - H04L 41/0895: Configuration of virtualised networks or elements, e.g. virtualised network function or OpenFlow elements - G06F 9/547: {Remote procedure calls [RPC]; Web services} - G06F 2009/45591: {Monitoring or debugging support} - G06F 2009/45595: {Network integration; Enabling network access in virtual machine instances} - G06F 2209/501: Performance criteria - G06F 2209/503: Resource availability - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Systems and methods are provided for servicing inference request by one of multiple machine learning models attached to a deployment cluster. The API server of a deployment cluster is not tightly coupled to any of multiple machine learning models attached to the deployment cluster. Upon receiving an inference request, the deployment cluster can retrieve the configuration parameters, including serialization formatting, for a target model identified in the inference request. The deployment cluster can utilize the retrieved parameters to service the inference request and return the results to a business system application. ## Claims ### Claim 1 A management system comprising: one or more physical or virtualized computing devices communicatively coupled to a business system application and a plurality of deployment clusters; a processor; and a non-transitory computer-readable medium configured to execute instructions by the processor to cause the management system to: determine model-specific configuration parameters for a machine learning model; create a model-specific entry in a data structure associated with the machine learning model; identify a deployment cluster of the plurality of deployment clusters to attach the machine learning model; and transmit the model-specific configuration parameters to the identified deployment cluster. ### Claim 2 (depends on 1) The management cluster of claim 1, wherein each deployment cluster of the plurality of deployment clusters comprises one or more virtual nodes of one or more host systems. ### Claim 3 (depends on 2) The management cluster of claim 2, wherein the management system and at least one deployment cluster are deployed on a same host system. ### Claim 4 (depends on 1) The management cluster of claim 1, wherein the data structure comprises at least one data structure including an entry for each machine learning model of a plurality of machine learning models registered with the management system. ### Claim 5 (depends on 4) The management cluster of claim 4, wherein the data structure comprises one of tables, linked lists, data trees, or a combination thereof. ### Claim 6 (depends on 1) The management cluster of claim 1, wherein identifying the deployment cluster comprises determining a compute resources requirement for the machine learning model. ### Claim 7 (depends on 6) The management cluster of claim 6, wherein, in response to identifying a deployment cluster of the plurality of deployment clusters has available compute resources compared to the compute resources requirement, attaching the machine learning model to the identified deployment cluster. ### Claim 8 (depends on 6) The management cluster of claim 6, wherein, in response to identifying no deployment cluster of the plurality of deployment clusters having available compute resources compared to the compute resources requirement, deploying a new deployment cluster comprising one or more virtual nodes and attaching the machine learning model to the new deployment cluster. ### Claim 9 A system comprising: a business system application encoded as instructions on a non-transitory computer-readable medium and executed on a processor; a plurality of deployment clusters comprising one or more physical or virtualized computing devices; and a management system comprising one or more physical or virtualized computing devices, communicatively coupled to the business system application and the plurality of deployment clusters, wherein the management system is configured to: determine model-specific configuration parameters for a machine learning model; create a model-specific entry in a data structure associated with the machine learning model; identify a deployment cluster of the plurality of deployment clusters to attach the machine learning model; and transmit the model-specific configuration parameters to the identified deployment cluster. ### Claim 10 (depends on 9) The system of claim 9, wherein each deployment cluster of the plurality of deployment clusters comprises one or more virtual nodes of one or more host systems. ### Claim 11 (depends on 10) The system of claim 10, wherein the management system and at least one deployment cluster are deployed on a same host system. ### Claim 12 (depends on 9) The system of claim 9, wherein the data structure comprises at least one data structure including an entry for each machine learning model of a plurality of machine learning models registered with the management system. ### Claim 13 (depends on 12) The system of claim 12, wherein the data structure comprises one of tables, linked lists, data trees, or a combination thereof. ### Claim 14 (depends on 9) The system of claim 9, wherein identifying the deployment cluster comprises determining a compute resources requirement for the machine learning model. ### Claim 15 (depends on 14) The system of claim 14, wherein, in response to identifying a deployment cluster of the plurality of deployment clusters has available compute resources compared to the compute resources requirement, attaching the machine learning model to the identified deployment cluster. ### Claim 16 (depends on 14) The system of claim 14, wherein, in response to identifying no deployment cluster of the plurality of deployment clusters having available compute resources compared to the compute resources requirement, deploying a new deployment cluster comprising one or more virtual nodes and attaching the machine learning model to the new deployment cluster. ### Claim 17 A non-transitory machine-readable storage medium storing instructions thereon that, when executed by a processor, cause the processor to: determine model-specific configuration parameters for a machine learning model; create a model-specific entry in a data structure associated with the machine learning model; identify a deployment cluster of a plurality of deployment clusters to attach the machine learning model; and transmit the model-specific configuration parameters to the identified deployment cluster. ### Claim 18 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein each deployment cluster of the plurality of deployment clusters comprises one or more virtual nodes of one or more host systems. ### Claim 19 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein a management system and at least one deployment cluster are deployed on a same host system. ### Claim 20 (depends on 17) The non-transitory machine-readable storage medium of claim 17, wherein identifying the deployment cluster comprises determining a compute resources requirement for the machine learning model.