# US 12,602,602 B2 · Systems and methods for validating forecasting machine learning models > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,602 B2 - Document ID: US12602602B2 - Status: Granted - Kind: B2 - Application number: 17/304,415 - Filed: 2021-06-21 - Published: 2022-12-22 - Granted: 2026-04-14 - Earliest claimed: 2021-06-21 - Assignees: Verizon Patent and Licensing Inc. - Applicants: Verizon Patent and Licensing Inc. - Inventors: Kushal Singla; Subham Biswas - Web page: https://patentgrep.com/patent/US12602602B2 ## CPC - G06N 5/04: Inference or reasoning models - G06N 20/00: Machine learning - G06Q 10/04: Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" (market predictions or forecasting for commercial activities G06Q30/0202) - G06Q 30/016: After-sales - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06Q: INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR ## Abstract A device may receive historical time series data and output data associated with a forecasting model and may process the historical time series data and the output data, with a proxy regression model, to determine inference data. The device may create perturbed data from the historical time series data, the output data, and the inference data, and may process the perturbed data, with the proxy regression model, to generate labelled data and to identify top features of the labelled data. The device may process subsets of the top features of the labelled data, with the proxy regression model, to determine feature data identifying an importance of each of the subsets of the top features, and may evaluate the proxy regression model and the feature data to calculate validation data for validating the forecasting model. The device may validate the forecasting model with the validation data. ## Claims ### Claim 1 A method, comprising: receiving, by a device, historical time series data and output data associated with a forecasting model; training, by the device, a proxy regression model with the historical time series data and the output data, wherein the trained proxy regression model outputs inference data identifying an importance of each time step of the historical time series data; analyzing, by the device, the historical time series data, the output data, and the inference data to identify confidential information in the historical time series data, the output data, and the inference data; creating, by the device, perturbed data from the historical time series data, the output data, and the inference data, wherein the perturbed data comprises noisy or synthetic data that protects the confidential information in the historical time series data, the output data, and the inference data; processing, by the device, the perturbed data, with the trained proxy regression model, to generate labelled data and to identify top features of the labelled data relative to other features of the labelled data; processing, by the device, subsets of the top features of the labelled data, with the trained proxy regression model, to determine feature data identifying an importance of each of the subsets of the top features of the labelled data; determining, by the device and based on the feature data, a plurality of types of error values associated with the trained proxy regression model; calculating validation data for validating the forecasting model based on the plurality of types of error values associated with the trained proxy regression model; validating, by the device, the forecasting model with the validation data; and retraining, by the device, the trained proxy regression model with the validation data. ### Claim 2 (depends on 1) The method of claim 1, wherein the historical time series data includes univariate time series data. ### Claim 3 (depends on 1) The method of claim 1, wherein the trained proxy regression model includes one or more of: a linear regression model, a random forest model, a logistic regression model, a ridge regression model, a lasso regression model, or a polynomial regression model. ### Claim 4 (depends on 1) The method of claim 1, wherein the plurality of types of error values associated with the trained proxy regression model comprise: a mean absolute error value associated with the trained proxy regression model and based on the feature data; a normalized mean absolute error percentage value associated with the trained proxy regression model and based on the feature data; and a mean absolute percentage error value associated with the trained proxy regression model and based on the feature data. ### Claim 5 (depends on 1) The method of claim 1, wherein the forecasting model is to generate one or more of: a network load prediction for operation of a network, or a call center staffing prediction for a customer service center. ### Claim 6 (depends on 1) The method of claim 1, further comprising: debugging the forecasting model based on the validation data and to generate a debugged forecasting model; and causing the debugged forecasting model to be implemented. ### Claim 7 A device, comprising: one or more processors configured to: receive historical time series data and output data associated with a forecasting model, wherein the historical time series data includes univariate time series data; train a proxy regression model with the historical time series data and the output data, wherein the trained proxy regression model outputs inference data identifying an importance of each time step of the historical time series data; analyze the historical time series data, the output data, and the inference data to identify confidential information in the historical time series data, the output data, and the inference data; create perturbed data from the historical time series data, the output data, and the inference data, wherein the perturbed data comprises noisy or synthetic data that protects the confidential information in the historical time series data, the output data, and the inference data; process the perturbed data, with the trained proxy regression model, to generate labelled data and to identify top features of the labelled data relative to other features of the labelled data; process subsets of the top features of the labelled data, with the trained proxy regression model, to determine feature data identifying an importance of each of the subsets of the top features of the labelled data; determine, based on the feature data, a plurality of types of error values associated with the trained proxy regression model; calculate validation data for validating the forecasting model based on the plurality of types of error values associated with the trained proxy regression model; validate the forecasting model with the validation data; and retrain the trained proxy regression model with the validation data. ### Claim 8 (depends on 7) The device of claim 7, wherein the one or more processors are further configured to: generate a recommended update to the forecasting model based on debugging the forecasting model with the validation data; and provide the recommended update for display. ### Claim 9 (depends on 7) The device of claim 7, wherein the one or more processors are further configured to: receive feedback on the validation data; and revalidate the forecasting model based on the feedback and the validation data. ### Claim 10 (depends on 7) The device of claim 7, wherein the one or more processors, to evaluate the trained proxy regression model and the feature data to calculate the validation data, are configured to: evaluate the trained proxy regression model and the feature data to calculate a dimension for the validation data. ### Claim 11 (depends on 7) The device of claim 7, wherein the historical time series data is a predictor for the trained proxy regression model and the output data is a label for the trained proxy regression model. ### Claim 12 A non-transitory computer-readable medium storing a set of instructions, the set of instructions comprising: one or more instructions that, when executed by one or more processors of a device, cause the device to: receive historical time series data and output data associated with a forecasting model; train a proxy regression model with the historical time series data and the output data, wherein the trained proxy regression model outputs inference data identifying an importance of each time step of the historical time series data; analyze the historical time series data, the output data, and the inference data to identify confidential information in the historical time series data, the output data, and the inference data; create perturbed data from the historical time series data, the output data, and the inference data, wherein the perturbed data comprises noisy or synthetic data that protects the confidential information in the historical time series data, the output data, and the inference data; process the perturbed data, with the trained proxy regression model, to generate labelled data and to identify top features of the labelled data relative to other features of the labelled data; process, by the device, subsets of the top features of the labelled data, with the trained proxy regression model, to determine feature data identifying an importance of each of the subsets of the top features of the labelled data; determine, by the device and based on the feature data, a plurality of types of error values associated with the trained proxy regression model; calculate validation data for validating the forecasting model based on the plurality of types of error values associated with the trained proxy regression model; validate the forecasting model with the validation data; and retrain the trained proxy regression model with the validation data. ### Claim 13 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the plurality of types of error values comprise: a mean absolute error value associated with the trained proxy regression model and based on the feature data; a normalized mean absolute error percentage value associated with the trained proxy regression model and based on the feature data; and a mean absolute percentage error value associated with the trained proxy regression model and based on the feature data. ### Claim 14 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the one or more instructions further cause the device to generate one or more of: a network load prediction for operation of a network, or a call center staffing prediction for a customer service center. ### Claim 15 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the one or more instructions further cause the device to: generate a recommended update to the forecasting model based on validating the forecasting model with the validation data; and provide the recommended update for display. ### Claim 16 (depends on 12) The non-transitory computer-readable medium of claim 12, wherein the one or more instructions further cause the device to: receive feedback on the validation data; and revalidate the forecasting model based on the feedback and the validation data. ### Claim 17 (depends on 1) The method of claim 1, wherein receiving the historical time series data and the output data comprises: receiving, from a network provider, the historical time series data and the output data. ### Claim 18 (depends on 1) The method of claim 1, further comprising: receiving, by the device, feedback on the validation data; and revalidating, by the device, the forecasting model based on the feedback and the validation data, wherein the forecasting model is a machine learning model.