# US 2021/0174246 A1 · Adaptive learning system utilizing reinforcement learning to tune hyperparameters in machine learning techniques

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174246 A1
- Document ID: US20210174246A1
- Status: Published application
- Kind: A1
- Application number: 16/707,694
- Filed: 2019-12-09
- Published: 2021-06-10
- Earliest claimed: 2019-12-09
- Assignees: Ciena Corporation
- Applicants: Ciena Corporation
- Inventors: Thomas Triplet
- Web page: https://patentgrep.com/patent/US20210174246A1

## CPC

- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 20/00: Machine learning
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/092: Reinforcement learning
- G06N 3/096: Transfer learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/20: Ensemble learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods are provided in the field of Artificial Intelligence (AI) for enhancing, improving, augmenting, or tuning hyperparameters of Machine Learning (ML) techniques for creating a ML model. According to one implementation, a ML method comprises a step of using Reinforcement Learning (RL) to tune hyperparameters of one or more ML techniques. The method also includes the step of training a ML model using the one or more ML techniques in which the respective hyperparameters were tuned in the RL.

## Claims

### Claim 1

A Machine Learning (ML) system comprising:
a processing device; and
a memory device configured to store a retrospect learning module having logic instructions configured to cause the processing device to
use Reinforcement Learning (RL) to tune hyperparameters of one or more ML techniques, and
train a ML model using the one or more ML techniques in which the respective hyperparameters were tuned with the RL.

### Claim 2 (depends on 1)

The ML system of claim 1, wherein the logic instructions further cause the processing device to
store information from one or more previous iterations of ML model-building processes, and
utilizing the stored information as a reward within the RL.

### Claim 3 (depends on 2)

The ML system of claim 2, wherein the stored information includes metrics of one or more intermediate ML models obtained during the one or more previous iterations, wherein the metrics include one or more of accuracy, precision, recall, a training time, an inference time, and a forgetting score, and wherein the forgetting score is used to evaluate how well the ML model-building processes can learn new patterns while retaining knowledge of previously learned patterns.

### Claim 4 (depends on 3)

The ML system of claim 3, wherein the logic instructions further cause the processing device to calculate the forgetting score by
using a first dataset to train a first model,
determining a first accuracy of the first model when applied to the first dataset,
using a second dataset to tune the first model to achieve a second model,
determining a second accuracy of the second model when applied to the second dataset,
determining a third accuracy of the second model when applied to the first dataset, and
calculating a ratio between the second accuracy and the third accuracy.

### Claim 5 (depends on 1)

The ML system of claim 1, wherein the logic instructions further cause the processing device to
receive an input dataset with respect to an environment for which the ML model is to be modeled,
split the input dataset into at least a training dataset and a testing dataset,
use the training dataset to build an intermediate ML model, and
use the testing dataset to obtain metrics about the intermediate ML model.

### Claim 6 (depends on 1)

The ML system of claim 1, wherein the retrospect learning module comprises
a dataset splitting module configured to split an input dataset from an environment in which a ML model is intended to operate,
a model building module configured to build ML models in multiple iterations,
a result testing module configured to obtain metrics regarding each iteration,
an automatic hyperparameter enhancement module configured to automatically tune the hyperparameters of ML techniques of the ML model, and
a tuning module configured to tune the ML model based on the tuned hyperparameters.

### Claim 7 (depends on 6)

The ML system of claim 6, wherein the retrospect learning module further comprises
a forgetting score calculating module for calculating a forgetting score used to evaluate how well the ML model can learn new patterns while retaining information about previously learned patterns.

### Claim 8

A method comprising the steps of:
using Reinforcement Learning (RL) to tune hyperparameters of one or more Machine Learning (ML) techniques; and
training a ML model using the one or more ML techniques in which the respective hyperparameters were tuned with the RL.

### Claim 9 (depends on 8)

The method of claim 8, wherein the step of using the RL-based system includes the steps of:
storing information from one or more previous iterations of ML model-building processes; and
utilizing the stored information as a reward within the RL-based system.

### Claim 10 (depends on 9)

The method of claim 9, wherein the stored information includes metrics of one or more intermediate ML models obtained during the one or more previous iterations, and wherein the metrics include one or more of accuracy, precision, recall, training time, inference time, and forgetting score.

### Claim 11 (depends on 10)

The method of claim 10, wherein the metrics include at least the forgetting score, and wherein the forgetting score is used to evaluate how well the ML model-building processes can learn new patterns while retaining knowledge of previously learned patterns.

### Claim 12 (depends on 11)

The method of claim 11, further comprising the step of calculating the forgetting score by:
using a first dataset to train a first model;
determining a first accuracy of the first model when applied to the first dataset;
using a second dataset to tune the first model to achieve a second model;
determining a second accuracy of the second model when applied to the second dataset;
determining a third accuracy of the second model when applied to the first dataset; and
calculating a ratio between the second accuracy and the third accuracy.

### Claim 13 (depends on 8)

The method of claim 8, further comprising the steps of:
receiving an input dataset with respect to an environment for which the ML model is to be modeled;
splitting the input dataset into at least a training dataset and a testing dataset;
using the training dataset to build an intermediate ML model; and
using the testing dataset to obtain metrics about the intermediate ML model.

### Claim 14 (depends on 13)

The method of claim 13, wherein the step of the splitting the input dataset further includes the step of the splitting the input dataset into the training dataset, the testing dataset, and a validation dataset, wherein the method further comprises the step of utilizing the validation dataset to perform cross-validation multiple times to evaluate the intermediate ML model during multiple iterations.

### Claim 15 (depends on 8)

The method of claim 8, wherein the RL-based system includes:
states defined as one or more of performance metrics, parameters of previously-training ML models, information provided by a human expert, information provided by an environment in which the ML model is intended to operate, and statistics about historical changes;
actions defined as a tuning of the hyperparameters; and
rewards defined as one or more of maximizing accuracy, precision, and recall; minimizing amount of data required; minimizing computation time; minimize human labelling; minimizing cost associated with large hyperparameter changes; maximizing transfer efficiency; minimizing forgetting score; and a configurable weighted combination of a plurality of these rewards.

### Claim 16

A non-transitory computer-readable medium configured to store computer logic having instructions that, when executed, cause one or more processing devices to:
use Reinforcement Learning (RL) to tune hyperparameters of one or more Machine Learning (ML) techniques; and
train a ML model using the one or more ML techniques in which the respective hyperparameters were tuned with the RL.

### Claim 17 (depends on 16)

The non-transitory computer-readable medium of claim 16, wherein the instructions further cause the one or more processing devices to
store information from one or more previous iterations of ML model-building processes, and
utilize the stored information as a reward within the RL-based system.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the stored information includes metrics of one or more intermediate ML models obtained during the one or more previous iterations, and wherein the metrics include one or more of accuracy, precision, recall, a training time, an inference time, and a forgetting score.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the instructions further cause the one or more processing devices to calculate the forgetting score to evaluate how well the ML model-building processes can learn new patterns while retaining knowledge of previously learned patterns.

### Claim 20 (depends on 19)

The non-transitory computer-readable medium of claim 19, wherein the instructions further cause the one or more processing devices to calculate the forgetting score by using a first dataset to train a first model,
determining a first accuracy of the first model when applied to the first dataset,
using a second dataset to tune the first model to achieve a second model,
determining a second accuracy of the second model when applied to the second dataset,
determining a third accuracy of the second model when applied to the first dataset, and
calculating a ratio between the second accuracy and the third accuracy.
