# US 12,216,738 B2 · Predicting performance of machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,216,738 B2 - Document ID: US12216738B2 - Status: Granted - Kind: B2 - Application number: 17/068,226 - Filed: 2020-10-12 - Published: 2022-04-14 - Granted: 2025-02-04 - Earliest claimed: 2020-10-12 - Assignees: International Business Machines Corporation - Applicants: International Business Machines Corporation - Inventors: Lukasz G. Cmielowski; Rafal Bigaj; Wojciech Sobala; Maksymilian Erazmus - Web page: https://patentgrep.com/patent/US12216738B2 ## CPC - G06F 18/2185: {the supervisor being an automated module, e.g. intelligent oracle} - G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting - G06N 3/0499: Feedforward networks - G06N 3/09: Supervised learning - G06N 20/00: Machine learning - G06Q 10/06393: {Score-carding, benchmarking or key performance indicator [KPI] analysis} - G06N 3/08: Learning methods - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract A computer-implemented method and computer program product for predicting an impact of an adjustment to a machine learning model to key performance indicators, and a forecasting engine. The computer-implemented method may comprise receiving a proposed adjustment to a machine learning model, calculating, using a regression machine learning model to ingest the proposed adjustment, a set of value components for a key performance indicator (KPI), calculating a plurality of results for the KPI using the set of value components, automatically determining whether the plurality of results exceeds a performance threshold, and recommending the proposed adjustment based on the determination. ## Claims ### Claim 1 A computer-implemented method for predicting an impact of an adjustment to a machine learning model to key performance indicators, the method comprising: receiving a proposed adjustment to a machine learning model; calculating, using a regression machine learning model to ingest the proposed adjustment, a set of value components for a key performance indicator (KPI) as indicator values using input data on a specified schedule; mapping the calculated indicator values onto scoring payload data; calculating a plurality of results for the KPI using the set of value components; automatically determining whether the plurality of results exceeds a performance threshold; recommending the proposed adjustment based on the determination; and training the regression model by iteratively performing: receiving a model scoring payload, an input data set, and a target data set; calculating a gradient that is a difference between an input data value of the input data set and a target data value of the target data set; and propagating the gradient through layers of the regression model to update synaptic weights of the regression model. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, further comprising analyzing a set of machine learning model metrics against a plurality of key performance indicators (KPIs). ### Claim 3 (depends on 1) The computer-implemented method of claim 1, further comprising calculating metric values for the proposed adjustment using model scoring payload data slices. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein determining whether the plurality of results exceeds the performance threshold comprises: deploying the proposed adjustment in a test environment; defining an expected model quality parameter; and monitoring the proposed adjustment in the test environment to produce test data; and recommending the proposed adjustment to a user when the test data exceeds the expected model quality parameter. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, further comprising approving the proposed adjustment based on the recommendation. ### Claim 6 A computer program product for predicting an impact of an adjustment to a machine learning model to key performance indicators, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to: receive a proposed new version of a machine learning model; calculate, using a regression machine learning model to ingest the proposed new version, a set of value components for a plurality of key performance indicators as indicator values using input data on a specified schedule; mapping the calculated indicator values onto scoring payload data; calculate a plurality of results for the plurality of key performance indicators, using the set of value components; automatically determine whether the plurality of results exceeds a performance threshold; recommend the proposed adjustment based on the determination; and train the regression model by the instructions causing the processor to iteratively perform operations to: receive a model scoring payload, an input data set, and a target data set; calculate a gradient that is a difference between an input data value of the input data set and a target data value of the target data set; and propagate the gradient through layers of the regression model to update synaptic weights of the regression model. ### Claim 7 (depends on 6) The computer program product of claim 6, further comprising program instructions to analyze a set of machine learning model metrics against the plurality of key performance indicators. ### Claim 8 (depends on 6) The computer program product of claim 6, further comprising program instructions to calculate metric values for the proposed new version using model scoring payload data slices. ### Claim 9 (depends on 6) The computer program product of claim 6, wherein determining whether the plurality of results exceeds the performance threshold comprises: deploying the proposed new version in a test environment; defining an expected model quality parameter; and monitoring the proposed new version in the test environment to produce test data; and recommending the proposed new version to a user when the test data exceeds the expected model quality parameter. ### Claim 10 A forecasting engine for a machine learning model, comprising: one or more processors coupled to one or more memories, the one or more memories comprising: an original machine learning model having performance statistics associated therewith; performance data for the original machine learning model; and program instructions that, when executed on the one or more processors, cause the one or more processors to: receive a proposed adjustment to the original machine learning model; calculate, using a regression machine learning model to ingest the proposed new version, a set of value components for a plurality of key performance indicators as indicator values using input data on a specified schedule; mapping the calculated indicator values onto scoring payload data; calculate a plurality of results for the plurality of key performance indicators, using the set of value components; automatically determine whether the plurality of results exceeds a performance threshold; recommend the proposed adjustment based on the determination; and train the regression model by the instructions causing the processor to iteratively perform operations to: receive a model scoring payload, an input data set, and a target data set; calculate a gradient that is a difference between an input data value of the input data set and a target data value of the target data set; and propagate the gradient through layers of the regression model to update synaptic weights of the regression model. ### Claim 11 (depends on 10) The forecasting engine of claim 10, further comprising program instructions executable by a processor to cause the processor to analyze a set of machine learning model metrics against a plurality of key performance indicators. ### Claim 12 (depends on 10) The forecasting engine of claim 10, further comprising program instructions to calculate metric values for the proposed adjustment using model scoring payload data slices. ### Claim 13 (depends on 10) The forecasting engine of claim 10, further comprising program instructions to automatically deploy the proposed adjustment based on the determining.