# US 11,310,141 B2 · Anomaly detection of model performance in an MLOps platform > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,310,141 B2 - Document ID: US11310141B2 - Status: Granted - Kind: B2 - Application number: 16/710,836 - Filed: 2019-12-11 - Published: 2021-06-17 - Granted: 2022-04-19 - Earliest claimed: 2019-12-11 - Assignees: Cisco Technology, Inc. - Applicants: Cisco Technology, Inc. - Inventors: Vinay Kumar Kolar; Jean-Philippe Vasseur; Grégory Mermoud; Pierre-Andre Savalle - Web page: https://patentgrep.com/patent/US11310141B2 ## CPC - H04L 43/14: {using software, i.e. software packages (network security related monitoring H04L63/1408)} - G06F 17/18: for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} - G06F 18/24: Classification techniques - G06N 20/00: Machine learning - H04L 41/16: using machine learning or artificial intelligence - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract In one embodiment, a service tracks performance of a machine learning model over time. The machine learning model is used to monitor one or more computer networks based on data collected from the one or more computer networks. The service also tracks performance metrics associated with training of the machine learning model. The service determines that a degradation of the performance of the machine learning model is anomalous, based on the tracked performance of the machine learning model and performance metrics associated with training of the model. The service initiates a corrective measure for the degradation of the performance, in response to determining that the degradation of the performance is anomalous. ## Claims ### Claim 1 A method, comprising: tracking, by a service, performance of a machine learning model over time, wherein the machine learning model is used to monitor one or more computer networks based on data collected from the one or more computer networks; tracking, by the service, performance metrics associated with training of the machine learning model, wherein the performance metrics are indicative of accuracy of the machine learning model as observed during training and testing prior to deployment; tracking, by the service, changes in the data collected from the one or more networks; determining, by the service, that a degradation of the performance of the machine learning model is anomalous by correlating the performance of the machine learning model over time with the performance metrics associated with training of the machine learning model and the tracked changes in the data collected from the one or more networks; and initiating, by the service, a corrective measure for the degradation of the performance, in response to determining that the degradation of the performance is anomalous. ### Claim 2 (depends on 1) The method as in claim 1, wherein the corrective measure comprises at least one of: pausing use of the machine learning model to monitor the one or more computer networks or retraining the machine learning model. ### Claim 3 (depends on 1) The method as in claim 1, wherein determining that the degradation of the performance of the machine learning model is anomalous comprises: detecting an anomaly in a correlation between the tracked performance of the machine learning model and the performance metrics associated with training of the machine learning model. ### Claim 4 (depends on 1) The method as in claim 1, wherein determining that the degradation of the performance of the machine learning model is anomalous comprises: determining whether a drift in the performance of the machine learning model between times at which the model the model is trained is anomalous. ### Claim 5 (depends on 1) The method as in claim 1, wherein the machine learning model comprises an anomaly detector, and wherein the tracked performance is indicative of a percentage of anomalies raised by the anomaly detector for the one or more networks. ### Claim 6 (depends on 1) The method as in claim 1, wherein the machine learning model comprises a classifier, and wherein the tracked performance is indicative of a detection probability of the classifier. ### Claim 7 (depends on 6) The method as in claim 6, wherein the classifier is configured to classify devices in the one or more networks by device type. ### Claim 8 (depends on 1) The method as in claim 1, wherein the machine learning model is configured to predict a tunnel failure in the one or more networks. ### Claim 9 (depends on 1) The method as in claim 1, further comprising: detecting a drift in the performance metrics associated with training the machine learning model. ### Claim 10 An apparatus, comprising: one or more network interfaces; a processor coupled to the network interfaces and configured to execute one or more processes; and a memory configured to store a process executable by the processor, the process when executed configured to: track performance of a machine learning model over time, wherein the machine learning model is used to monitor one or more computer networks based on data collected from the one or more computer networks; track performance metrics associated with training of the machine learning model, wherein the performance metrics are indicative of accuracy of the machine learning model as observed during training and testing prior to deployment; track changes in the data collected from the one or more networks; determine that a degradation of the performance of the machine learning model is anomalous by correlating the performance of the machine learning model over time with the performance metrics associated with training of the machine learning model and the tracked changes in the data collected from the one or more networks; and initiate a corrective measure for the degradation of the performance, in response to determining that the degradation of the performance is anomalous. ### Claim 11 (depends on 10) The apparatus as in claim 10, wherein the corrective measure comprises at least one of: pausing use of the machine learning model to monitor the one or more computer networks or retraining the machine learning model. ### Claim 12 (depends on 10) The apparatus as in claim 10, wherein the apparatus determines that the degradation of the performance of the machine learning model is anomalous by: detecting an anomaly in a correlation between the tracked performance of the machine learning model and the performance metrics associated with training of the machine learning model. ### Claim 13 (depends on 10) The apparatus as in claim 10, wherein the apparatus determines that the degradation of the performance of the machine learning model is anomalous by: determining whether a drift in the performance of the machine learning model between times at which the model the model is trained is anomalous. ### Claim 14 (depends on 10) The apparatus as in claim 10, wherein the process when executed is further configured to: detect a drift in the performance metrics associated with training the machine learning model. ### Claim 15 (depends on 10) The apparatus as in claim 10, wherein the machine learning model is configured to detect a problem in a wireless network. ### Claim 16 A tangible, non-transitory, computer-readable medium storing program instructions that cause a service to execute a process comprising: tracking, by the service, performance of a machine learning model over time, wherein the machine learning model is used to monitor one or more computer networks based on data collected from the one or more computer networks; tracking, by the service, performance metrics associated with training of the machine learning model, wherein the performance metrics are indicative of accuracy of the machine learning model as observed during training and testing prior to deployment; tracking, by the service, changes in the data collected from the one or more networks; determining, by the service, that a degradation of the performance of the machine learning model is anomalous by correlating the performance of the machine learning model over time with the performance metrics associated with training of the machine learning model and the tracked changes in the data collected from the one or more networks; and initiating, by the service, a corrective measure for the degradation of the performance, in response to determining that the degradation of the performance is anomalous.