# US 2021/0344745 A1 · Adaptive training of machine learning models based on live performance metrics > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2021/0344745 A1 - Document ID: US20210344745A1 - Status: Published application - Kind: A1 - Application number: 16/865,517 - Filed: 2020-05-04 - Published: 2021-11-04 - Earliest claimed: 2020-05-04 - Assignees: Cisco Technology, Inc. - Applicants: Cisco Technology, Inc. - Inventors: Grégory Mermoud; Jean-Philippe Vasseur; Pierre-André Savalle; Vinay Kumar Kolar; David Tedaldi - Web page: https://patentgrep.com/patent/US20210344745A1 ## CPC - H04L 41/16: using machine learning or artificial intelligence - G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques} - G06F 18/285: {Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system} - G06N 3/045: Combinations of networks - G06N 3/08: Learning methods - G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning - G06N 3/09: Supervised learning - G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn - G06N 20/00: Machine learning - G06N 20/20: Ensemble learning - H04L 41/142: using statistical or mathematical methods - H04W 4/02: Services making use of location information - H04L 41/5009: Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF] - H04L 41/5067: Customer-centric QoS measurements - H04L 43/0888: Throughput - H04L 67/10: in which an application is distributed across nodes in the network (software deployment G06F8/60; multiprogramming arrangements G06F9/46) - H04W 12/02: Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII] - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72) ## Abstract In one embodiment, a device deploys a first machine learning model to an inference location in a network. The first machine learning model is used at the inference location to make inferences about the network. The device receives, from the inference location, an indication that the first machine learning model is exhibiting poor performance. The device identifies a corrective measure for the poor performance that minimizes resource consumption by a model training pipeline of the device. The device deploys, based on the corrective measure, a second machine learning model to the inference location. The second machine learning model is used in lieu of the first machine learning model to make the inferences about the network. ## Claims ### Claim 1 A method comprising: deploying, by a device, a first machine learning model to an inference location in a network, wherein the first machine learning model is used at the inference location to make inferences about the network; receiving, at a device and from the inference location, an indication that the first machine learning model is exhibiting poor performance; identifying, by the device, a corrective measure for the poor performance that minimizes resource consumption by a model training pipeline of the device; and deploying, by the device and based on the corrective measure, a second machine learning model to the inference location, wherein the second machine learning model is ii used in lieu of the first machine learning model to make the inferences about the network. ### Claim 2 (depends on 1) The method as in claim 1, wherein the inference location is a router, switch, or wireless access point controller in the network. ### Claim 3 (depends on 1) The method as in claim 1, wherein the indication further indicates a shifted input data distribution for the first machine learning model, and wherein identifying the corrective measure for the poor performance comprises: selecting the second machine learning model for deployment to the inference s location from among a plurality of existing machine learning models, based on the second machine learning model being trained on an input data distribution that matches the shifted input data distribution for the first machine learning model. ### Claim 4 (depends on 1) The method as in claim 1, wherein the corrective measure comprises incrementally retraining the first machine learning model, the method further comprising: generating the second machine learning model by incrementally retraining the first machine learning model using more recent data than was initially used to train the first machine learning model. ### Claim 5 (depends on 1) The method as in claim 1, further comprising: identifying, by the device and using a dependency graph, one or more other machine learning models used at the inference location that depend on an output of the first machine learning model; and proactively sending, by the device and in response to the indication that the first machine learning model is exhibiting poor performance, one or more replacement machine learning models to the inference location to be used in lieu of the one or more other machine learning models that depend on the output of the first machine learning model. ### Claim 6 (depends on 1) The method as in claim 1, wherein the corrective measure comprises rescaling input features of the first machine learning model, the method further comprising: generating the second machine learning model by rescaling the input features of the first machine learning model. ### Claim 7 (depends on 1) The method as in claim 1, wherein the corrective measure comprises generating a new machine learning model, the method further comprising: generating a plurality of machine learning models, based on a resource consumption budget; and selecting the second machine learning model from among the plurality of machine learning models for deployment to the inference location. ### Claim 8 (depends on 1) The method as in claim 1, wherein the corrective measure is identified based in part on a number of inference locations to which the first machine learning model was deployed that reported that the first machine learning model exhibited poor performance. ### Claim 9 (depends on 1) The method as in claim 1, wherein the inferences about the network comprise predicted failures in the network. ### Claim 10 An apparatus, comprising: one or more network interfaces; a processor coupled to the one or more network interfaces; and a memory configured to store a process that is executable by the processor, the process when executed configured to: deploy a first machine learning model to an inference location in a network, wherein the first machine learning model is used at the inference location to make inferences about the network; receive, from the inference location, an indication that the first machine learning model is exhibiting poor performance; identify a corrective measure for the poor performance that minimizes resource consumption by a model training pipeline of the apparatus; and deploy, based on the corrective measure, a second machine learning model to the inference location, wherein the second machine learning model is used in is lieu of the first machine learning model to make the inferences about the network. ### Claim 11 (depends on 10) The apparatus as in claim 10, wherein the inference location is a router, switch, or wireless access point controller in the network. ### Claim 12 (depends on 10) The apparatus as in claim 10, wherein the indication further indicates a shifted input data distribution for the first machine learning model, and wherein the apparatus indicates the corrective measure for the poor performance by: selecting the second machine learning model for deployment to the inference location from among a plurality of existing machine learning models, based on the second machine learning model being trained on an input data distribution that matches the shifted input data distribution for the first machine learning model. ### Claim 13 (depends on 10) The apparatus as in claim 10, wherein the corrective measure comprises incrementally retraining the first machine learning model, the process when executed further configured to: generate the second machine learning model by incrementally retraining the first machine learning model using more recent data than was initially used to train the first machine learning model. ### Claim 14 (depends on 10) The apparatus as in claim 10, wherein the process when executed is further configured to: identify, using a dependency graph, one or more other machine learning models used at the inference location that depend on an output of the first machine learning model; and proactively send, in response to the indication that the first machine learning model is exhibiting poor performance, one or more replacement machine learning models to the inference location to be used in lieu of the one or more other machine learning models that depend on the output of the first machine learning model. ### Claim 15 (depends on 10) The apparatus as in claim 10, wherein the corrective measure comprises rescaling input features of the first machine learning model, the process when executed further configured to: generate the second machine learning model by rescaling the input features of the first machine learning model. ### Claim 16 (depends on 10) The apparatus as in claim 10, wherein the corrective measure comprises generating a new machine learning model, the process when executed further configured to: generate a plurality of machine learning models, based on a resource consumption budget; and select the second machine learning model from among the plurality of machine learning models for deployment to the inference location. ### Claim 17 (depends on 10) The apparatus as in claim 10, wherein the corrective measure is identified based in part on a number of inference locations to which the first machine learning model was deployed that reported that the first machine learning model exhibited poor performance. ### Claim 18 (depends on 10) The apparatus as in claim 10, wherein the inferences about the network comprise predicted failures in the network. ### Claim 19 A tangible, non-transitory, computer-readable medium storing program instructions that cause a device to execute a process comprising: deploying, by the device, a first machine learning model to an inference location in a network, wherein the first machine learning model is used at the inference location to make inferences about the network; receiving, at a device and from the inference location, an indication that the first machine learning model is exhibiting poor performance; identifying, by the device, a corrective measure for the poor performance that minimizes resource consumption by a model training pipeline of the device; and deploying, by the device and based on the corrective measure, a second machine learning model to the inference location, wherein the second machine learning model is used in lieu of the first machine learning model to make the inferences about the network. ### Claim 20 (depends on 19) The computer-readable medium as in claim 19, wherein the process further comprises: identifying, by the device and using a dependency graph, one or more other machine learning models used at the inference location that depend on an output of the first machine learning model; and proactively sending, by the device and in response to the indication that the first machine learning model is exhibiting poor performance, one or more replacement machine learning models to the inference location to be used in lieu of the one or more other machine learning models that depend on the output of the first machine learning model.