# US 2019/0095818 A1 · Gradient-based auto-tuning for machine learning and deep learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0095818 A1
- Document ID: US20190095818A1
- Status: Published application
- Kind: A1
- Application number: 15/885,515
- Filed: 2018-01-31
- Published: 2019-03-28
- Earliest claimed: 2017-09-28
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: Venkatanathan Varadarajan; Sam Idicula; Sandeep Agrawal; Nipun Agarwal
- Web page: https://patentgrep.com/patent/US20190095818A1

## CPC

- G06N 20/00: Machine learning
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 3/0985: Hyperparameter optimisation; Meta-learning; Learning-to-learn
- G06N 5/022: Knowledge engineering; Knowledge acquisition
- G06N 20/20: Ensemble learning
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Herein, horizontally scalable techniques efficiently configure machine learning algorithms for optimal accuracy and without informed inputs. In an embodiment, for each particular hyperparameter, and for each epoch, a computer processes the particular hyperparameter. An epoch explores one hyperparameter based on hyperparameter tuples. A respective score is calculated from each tuple. The tuple contains a distinct combination of values, each of which is contained in a value range of a distinct hyperparameter. All values of a tuple that belong to the particular hyperparameter are distinct. All values of a tuple that belong to other hyperparameters are held constant. The value range of the particular hyperparameter is narrowed based on an intersection point of a first line based on the scores and a second line based on the scores. A machine learning algorithm is optimally configured from repeatedly narrowed value ranges of hyperparameters. The configured algorithm is invoked to obtain a result.

## Claims

### Claim 1

A method comprising:
for each particular hyperparameter of a plurality of hyperparameters that is not a categorical hyperparameter, for each epoch in a sequence of epochs, processing the particular hyperparameter by:
a) for each hyperparameter tuple of a plurality of hyperparameter tuples, calculating a score based on the hyperparameter tuple, wherein:
the hyperparameter tuple contains a distinct plurality of values,
each value of the distinct plurality of values is contained in a current value range of a distinct hyperparameter of a plurality of hyperparameters, and
all values of the plurality of hyperparameter tuples that belong to a same hyperparameter have a same value unless said same hyperparameter is said particular hyperparameter; and
b) narrowing the current value range of the particular hyperparameter based on an intersection point of a first line that is based on said scores and a second line that is based on said scores;
configuring, based on the current value ranges of the plurality of hyperparameters, a machine learning algorithm;
invoking the machine learning algorithm to obtain a result.

### Claim 2 (depends on 1)

The method of claim 1 wherein said calculating said score comprises updating a best hyperparameter value and a best hyperparameter score that are associated with said particular hyperparameter when said score exceeds said best hyperparameter score.

### Claim 3 (depends on 2)

The method of claim 2 wherein said same value that belongs to said same hyperparameter is said best hyperparameter value that is associated with said particular hyperparameter in a previous epoch.

### Claim 4 (depends on 2)

The method of claim 2 further comprising, at the end of said each epoch:
calculating a score based on said best hyperparameter value that is associated with said each hyperparameter of said plurality of hyperparameters;
updating said best hyperparameter value and said best hyperparameter score that are associated with said each hyperparameter of said plurality of hyperparameters when said score exceeds said best hyperparameter score.

### Claim 5 (depends on 1)

The method of claim 1 wherein for each epoch in a sequence of epochs comprises, when an improvement of a best hyperparameter score of said epoch exceeds a threshold:
for each categorical hyperparameter of said plurality of hyperparameters that is a categorical hyperparameter, processing the particular hyperparameter by:
for each hyperparameter tuple of a second plurality of hyperparameter tuples, calculating a score based on the hyperparameter tuple, wherein:
the hyperparameter tuple contains a distinct plurality of values,
all values of the plurality of hyperparameter tuples that belong to a same hyperparameter have a same value unless said same hyperparameter is said categorical hyperparameter
all values of the plurality of hyperparameter tuples that belong to said categorical hyperparameter have a distinct value.

### Claim 6 (depends on 1)

The method of claim 1 wherein said for each particular hyperparameter occurs simultaneously for said plurality of hyperparameters.

### Claim 7 (depends on 1)

The method of claim 1 wherein:
said plurality of hyperparameter tuples comprises a first subset of hyperparameter tuples and a second subset of hyperparameter tuples;
all values of the first subset that belong to said particular hyperparameter are equally spaced within said current value range of said particular hyperparameter;
for each equally-spaced hyperparameter tuple of the first subset, the second subset contains colinear hyperparameter tuple that differs from the equally-spaced hyperparameter tuple only by a predefined amount from the value of the equally-spaced hyperparameter tuple that belongs to said particular hyperparameter.

### Claim 8 (depends on 7)

The method of claim 7 wherein:
the first line contains a first point and a second point;
the second line contains a third point and a fourth point;
the first point is based on a first hyperparameter tuple of the first subset that has a best score;
the second point is based on said colinear hyperparameter tuple for said first hyperparameter tuple;
the third point is based on a second hyperparameter tuple of the first subset that has a second best score;
the fourth point is based on said colinear hyperparameter tuple for said second hyperparameter tuple.

### Claim 9 (depends on 8)

The method of claim 8 wherein narrowing the current value range of the particular hyperparameter is further based on a particular point of the first line.

### Claim 10 (depends on 9)

The method of claim 9 wherein said particular point is one of: said first point or said second point.

### Claim 11 (depends on 9)

The method of claim 9 wherein based on an intersection point comprises based on a point that is between the intersection point and said third point.

### Claim 12 (depends on 1)

The method of claim 1 wherein:
said configuring based on the current value ranges of the plurality of hyperparameters comprises configuring based on a plurality of greedy values;
each greedy value of the plurality of greedy values belongs to a different hyperparameter of the plurality of hyperparameters;
each greedy value of the plurality of greedy values is calculated by a gradient ascent based on the value that belongs to said different hyperparameter in a hyperparameter tuple having the best of all scores.

### Claim 13 (depends on 1)

The method of claim 1 wherein said all values of the plurality of hyperparameter tuples that belong to said same hyperparameter have said same value initially comprises said same value is a maximum value of said current value range or a minimum value of said current value range.

### Claim 14 (depends on 1)

The method of claim 1 wherein said calculating a score for each hyperparameter tuple comprises simultaneously calculating scores for multiple hyperparameter tuples.

### Claim 15 (depends on 14)

The method of claim 14 wherein a count of tuples in said plurality of hyperparameter tuples is based on a count of processors.

### Claim 16 (depends on 1)

The method of claim 1 wherein said score based on the hyperparameter tuple comprises said score based on training said machine learning algorithm that is configured with the hyperparameter tuple.

### Claim 17 (depends on 16)

The method of claim 16 wherein said score is based on at least one of:
fitness of said machine learning algorithm during or after said training, or
time spent during said training.

### Claim 18 (depends on 16)

The method of claim 16 wherein said training is based on cross-validation comprising folds.

### Claim 19 (depends on 18)

The method of claim 18 wherein at least one of:
said score is based on an average score of said folds,
said folds consist of five folds, or
said folds are stratified.

### Claim 20 (depends on 1)

The method of claim 1 wherein said score is at least one of: a classification score, an F1 score, a score based on a loss metric.

### Claim 21 (depends on 1)

The method of claim 1 wherein calculating said score based on said hyperparameter tuple comprises:
using said hyperparameter tuple as a lookup key for retrieving a known score;
using the known score as said score if the known score is retrieved;
otherwise, calculating said score and associating said lookup key with said score.

### Claim 22 (depends on 1)

The method of claim 1 wherein said sequence of epochs for said particular hyperparameter terminates upon an occurrence of at least one of:
the first line does not intersect the second line;
a width of the current value range of the particular hyperparameter does not exceed a first threshold;
said intersection point indicates a value for the particular hyperparameter that is not within the current value range of the particular hyperparameter.

### Claim 23

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause:
for each particular hyperparameter of a plurality of hyperparameters that is not a categorical hyperparameter, for each epoch in a sequence of epochs, processing the particular hyperparameter by:
a) for each hyperparameter tuple of a plurality of hyperparameter tuples, calculating a score based on the hyperparameter tuple, wherein:
the hyperparameter tuple contains a distinct plurality of values,
each value of the distinct plurality of values is contained in a current value range of a distinct hyperparameter of a plurality of hyperparameters, and
all values of the plurality of hyperparameter tuples that belong to a same hyperparameter have a same value unless said same hyperparameter is said particular hyperparameter; and
b) narrowing the current value range of the particular hyperparameter based on an intersection point of a first line that is based on said scores and a second line that is based on said scores;
configuring, based on the current value ranges of the plurality of hyperparameters, a machine learning algorithm;
invoking the machine learning algorithm to obtain a result.
