# US 12,762,476 B2 · AI customized MLOps framework in a multi-tenant cloud environment > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,762,476 B2 - Document ID: US12762476B2 - Status: Granted - Kind: B2 - Application number: 18/110,074 - Filed: 2023-02-15 - Published: 2024-08-15 - Granted: 2026-10-13 - Earliest claimed: 2023-02-15 - Assignees: Kyndryl, Inc. - Applicants: Kyndryl, Inc. - Inventors: Omar Odibat; Umar Mohamed Iyoob Umar; Mouleswara Reddy Chintakunta; Manish Mitruka; Gail Camille Guerrero - Web page: https://patentgrep.com/patent/US12762476B2 ## CPC - G06N 20/00: Machine learning - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A computer implemented system, method, and computer program product are disclosed for managing a machine learning model operation (MLOps) for cost forecasting models. The MLOps receives a request for a subscription machine learning (ML) model by a tenant having a corresponding tenant configuration profile and automatically selects a subscription ML model in a model registry based on the tenant configuration profile. The MLOps deploys the selected subscription ML model to the tenant and monitors usage by the tenant of the currently operating ML model at a pre-determined refresh frequency. The MLOps determines whether the currently operating ML model exceeds a pre-determined accuracy threshold, and automatically deploys a second subscription ML model from the model registry to the tenant in place of the currently operating ML model in response to the pre-determined accuracy threshold being exceeded. ## Claims ### Claim 1 A computer implemented method, comprising: automatically selecting, by a multi-tenant cloud server, one of at least one subscription machine learning (ML) models in a model registry based on a tenant configuration profile of a tenant, responsive to receiving a request for the subscription ML model, wherein the tenant configuration profile includes information for data pull, data prep, feature engineering, model search, model validation and testing, and model evaluation associated with cloud service needs of the tenant; deploying, by the multi-tenant cloud server, the automatically selected subscription ML model to the tenant as a currently operating ML model; monitoring, by the multi-tenant cloud server, usage of the currently operating ML model at a pre-determined refresh frequency using a dynamic configuration machine learning model, the pre-determined refresh frequency based at least on the tenant configuration profile and a configuration profile category of the currently operating ML model; determining, by the multi-tenant cloud server, whether the currently operating ML model exceeds a pre-determined accuracy threshold; and automatically deploying, by the multi-tenant cloud server, a second subscription ML model from the model registry to the tenant in place of the currently operating ML model and becoming the currently operating ML model in response to the pre-determined accuracy threshold not being exceeded. ### Claim 2 (depends on 1) The computer implemented method as recited in claim 1, further comprising: receiving by the multi-tenant cloud server, the at least one subscription machine learning (ML) model; and storing, by the multi-tenant cloud server, the at least one subscription ML model in the model registry communicatively coupled to the multi-tenant cloud server, each of the at least one subscription ML models corresponding to a pre-determined configuration profile category, the pre-determined configuration profile category including an accuracy threshold and refresh frequency, wherein the pre-determined accuracy threshold is based at least on the accuracy threshold of the configuration profile category of the currently operating ML model. ### Claim 3 (depends on 1) The computer implemented method as recited in claim 1, further comprising: automatically updating, by the multi-tenant cloud server, the tenant configuration profile using the dynamic configuration machine learning model, based at least on one of model performance of the currently operating ML model, cost data of the tenant, resource constraints of the tenant, or a service level agreement between the tenant and the multi-tenant cloud server. ### Claim 4 (depends on 3) The computer implemented method as recited in claim 3, wherein automatically updating the tenant configuration profile further comprises performing at least one of: automatically modifying the accuracy threshold corresponding to the tenant and the currently operating ML model, and automatically modifying the pre-determined refresh frequency corresponding to the tenant and the currently operating ML model, in the tenant configuration profile. ### Claim 5 (depends on 1) The computer implemented method as recited in claim 1, wherein the automatically selecting one of the at least one subscription ML models in the model registry further comprises: selecting the one of the at least one subscription ML models using a dynamic model selector (DMS) machine learning (ML) model based at least on infrastructure of the tenant, low/medium/high profile feature of the tenant, tenant configuration profile and subscription ML model configuration category. ### Claim 6 (depends on 1) The computer implemented method as recited in claim 1, further comprising: automatically updating, by the multi-tenant cloud server, the configuration profile category of the currently running ML model based at least on historical model performance of the currently running ML model in one or more of a plurality of tenants using the currently running ML model. ### Claim 7 (depends on 1) The computer implemented method as recited in claim 1, wherein the pre-determined refresh frequency for the pre-determined configuration profile category of the currently running ML model is selected from the group of pre-determined refresh frequencies consisting of: hourly, daily, weekly, semi-weekly, bi weekly, monthly, semi-monthly, bi-monthly, annually, and semi-annually. ### Claim 8 (depends on 1) The computer implemented method as recited in claim 1, further comprising: continuing to monitor, by the multi-tenant cloud server, usage of the currently operating ML model at the pre-determined refresh frequency, the pre-determined refresh frequency based at least on the tenant configuration profile and the configuration profile category of the currently operating ML model; generating, by the multi-tenant cloud server, a new version of the currently running ML model based at least on historical model performance of the currently running ML model in one or more of a plurality of tenants using the currently running ML model; associating, by the multi-tenant cloud server, the new version of the currently running ML model with a new pre-determined configuration profile category; and storing, by the multi-tenant cloud server, the new version of the currently running ML model in the model registry as a default ML model for the new pre-determined configuration profile category. ### Claim 9 (depends on 1) The computer implemented method as recited in claim 1, wherein the subscription ML model is a cloud cost forecasting model. ### Claim 10 A computer program product comprising one or more computer readable storage media having program instructions collectively stored on the one or more computer readable storage media, the program instructions that when executed: automatically select one of at least one subscription machine learning (ML) models in a model registry based on a tenant configuration profile of a tenant, responsive to receiving a request for the subscription ML model, wherein the tenant configuration profile includes information associated with cloud service needs of the tenant; deploy the automatically selected subscription ML model to the tenant as a currently operating ML model; monitor usage of the currently operating ML model at a pre-determined refresh frequency using a dynamic configuration machine learning model, the pre-determined refresh frequency based at least on the tenant configuration profile and a configuration profile category of the currently operating ML model; determine whether the currently operating ML model exceeds a pre-determined accuracy threshold; and automatically deploy a second subscription ML model from the model registry to the tenant in place of the currently operating ML model and becoming the currently operating ML model in response to the pre-determined accuracy threshold not being exceeded. ### Claim 11 (depends on 10) The computer program product as recited in claim 10, further comprising program instructions that when executed: receive at least one subscription ML model; and store the at least one subscription ML model in the model registry communicatively coupled to a multi-tenant cloud server, each of the at least one subscription ML models corresponding to a pre-determined configuration profile category, the pre-determined configuration profile category including an accuracy threshold and refresh frequency, wherein the pre-determined accuracy threshold is based at least on the accuracy threshold of the configuration profile category of the currently operating ML model. ### Claim 12 (depends on 11) The computer program product as recited in claim 11, wherein automatically updating the tenant configuration profile further comprises program instructions, that when executed, perform at least one of: automatically modify the accuracy threshold corresponding to the tenant and the currently operating ML model, and automatically modify the pre-determined refresh frequency corresponding to the tenant and the currently operating ML model, in the tenant configuration profile. ### Claim 13 (depends on 10) The computer program product as recited in claim 10, further comprising program instructions that when executed: automatically update the tenant configuration profile using the dynamic configuration machine learning model, based at least on one of model performance of the currently operating ML model, cost data of the tenant, resource constraints of the tenant, or a service level agreement between the tenant and the multi-tenant cloud server. ### Claim 14 (depends on 10) The computer program product as recited in claim 10, wherein the automatically selecting one of the at least one subscription ML models in the model registry further comprises program instructions that when executed: select the one of the at least one subscription ML models using a dynamic model selector (DMS) machine learning (ML) model based at least on infrastructure of the tenant, low/medium/high profile feature of the tenant, tenant configuration profile and subscription ML model configuration category. ### Claim 15 (depends on 10) The computer program product as recited in claim 10, further comprising program instructions that when executed: automatically update the pre-determined configuration profile category of the currently running ML model based at least on historical model performance of the currently running ML model in one or more of a plurality of tenants using the currently running ML model. ### Claim 16 (depends on 10) The computer program product as recited in claim 10, wherein the pre-determined refresh frequency for the pre-determined configuration profile category of the currently running ML model is selected from the group of pre-determined refresh frequencies consisting of: hourly, daily, weekly, semi-weekly, bi weekly, monthly, semi-monthly, bi-monthly, annually, and semi-annually. ### Claim 17 (depends on 10) The computer program product as recited in claim 10, further comprising program instructions that when executed: continue to monitor usage of the currently operating ML model at the pre-determined refresh frequency, the pre-determined refresh frequency based at least on the tenant configuration profile and the configuration profile category of the currently operating ML model; generate a new version of the currently running ML model based at least on historical model performance of the currently running ML model in one or more of a plurality of tenants using the currently running ML model; associate the new version of the currently running ML model with a new pre-determined configuration profile category; and store the new version of the currently running ML model in the model registry as a default ML model for the new pre-determined configuration profile category. ### Claim 18 A system comprising: a processor, a computer readable memory, one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions that when executed: automatically select one of at least one subscription machine learning (ML) models in a model registry based on a tenant configuration profile of a tenant, responsive to receiving a request for the subscription ML model, wherein the tenant configuration profile includes information associated with cloud service needs of the tenant; deploy the automatically selected subscription ML model to the tenant as a currently operating ML model; monitor usage of the currently operating ML model at a pre-determined refresh frequency using a dynamic configuration machine learning model, the pre-determined refresh frequency based at least on the tenant configuration profile and a configuration profile category of the currently operating ML model; determine whether the currently operating ML model exceeds a pre-determined accuracy threshold; and automatically deploy a second subscription ML model from the model registry to the tenant in place of the currently operating ML model and becoming the currently operating ML model in response to the pre-determined accuracy threshold not being exceeded. ### Claim 19 (depends on 18) The system as recited in claim 18, further comprising program instructions that when executed: receive at least one subscription ML model; and store the at least one subscription ML model in the model registry communicatively coupled to a multi-tenant cloud server, each of the at least one subscription ML models corresponding to a pre-determined configuration profile category, the pre-determined configuration profile category including an accuracy threshold and refresh frequency, wherein the pre-determined accuracy threshold is based at least on the accuracy threshold of the configuration profile category of the currently operating ML model. ### Claim 20 (depends on 18) The system as recited in claim 18, further comprising program instructions that when executed: continue to monitor usage of the currently operating ML model at the pre-determined refresh frequency, the pre-determined refresh frequency based at least on the tenant configuration profile and the configuration profile category of the currently operating ML model; generate a new version of the currently running ML model based at least on historical model performance of the currently running ML model in one or more of a plurality of tenants using the currently running ML model; associate the new version of the currently running ML model with a new pre-determined configuration profile category; and store the new version of the currently running ML model in the model registry as a default ML model for the new pre-determined configuration profile category.