# US 2019/0095819 A1 · Scalable and efficient distributed auto-tuning of machine learning and deep learning models

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0095819 A1
- Document ID: US20190095819A1
- Status: Published application
- Kind: A1
- Application number: 16/137,719
- Filed: 2018-09-21
- Published: 2019-03-28
- Earliest claimed: 2017-09-27
- Assignees: Oracle International Corporation
- Applicants: Oracle International Corporation
- Inventors: VENKATANATHAN VARADARAJAN; SAM IDICULA; SANDEEP AGRAWAL; NIPUN AGARWAL
- Web page: https://patentgrep.com/patent/US20190095819A1

## CPC

- G06N 20/00: Machine learning
- G06F 9/4881: {Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues}
- G06F 9/505: {considering the load}
- G06N 3/048: Activation functions
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 5/01: Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/20: Ensemble learning
- G06N 3/126: Evolutionary algorithms, e.g. genetic algorithms or genetic programming
- G06N 20/10: using kernel methods, e.g. support vector machines [SVM]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Herein are techniques for automatic tuning of hyperparameters of machine learning algorithms. System throughput is maximized by horizontally scaling and asynchronously dispatching the configuration, training, and testing of an algorithm. In an embodiment, a computer stores a best cost achieved by executing a target model based on best values of the target algorithm's hyperparameters. The best values and their cost are updated by epochs that asynchronously execute. Each epoch has asynchronous costing tasks that explore a distinct hyperparameter. Each costing task has a sample of exploratory values that differs from the best values along the distinct hyperparameter. The asynchronous costing tasks of a same epoch have different values for the distinct hyperparameter, which accomplishes an exploration. In an embodiment, an excessive update of best values or best cost creates a major epoch for exploration in a subspace that is more or less unrelated to other epochs, thereby avoiding local optima.

## Claims

### Claim 1

A method comprising:
storing: a) a best cost achieved by executing a target machine learning (ML) model based on a best plurality of values respectively associated with a plurality of hyperparameters, and b) said best plurality of values;
asynchronously executing a plurality of epochs, wherein:
each epoch of the plurality of epochs is associated with a distinct hyperparameter of the plurality of hyperparameters;
each epoch of the plurality of epochs comprises a plurality of asynchronous costing tasks;
each costing task of the plurality of asynchronous costing tasks comprises an exploratory plurality of values respectively associated with the plurality of hyperparameters and based on said best plurality of values; and
the plurality of asynchronous costing tasks of said each epoch have different values for said distinct hyperparameter;
updating the best cost and the best plurality of values when said each costing task measures an actual cost for said exploratory plurality of values that is less than the best cost;
wherein the method is performed by one or more computers.

### Claim 2 (depends on 1)

The method of claim 1 wherein asynchronously executing said plurality of epochs comprises concurrently executing said plurality of epochs.

### Claim 3 (depends on 1)

The method of claim 1 further comprising canceling an epoch of the plurality of epochs that is selected based on at least one of: an age of the epoch, or an actual cost measured by a costing task of the plurality of asynchronous costing tasks of the epoch.

### Claim 4 (depends on 1)

The method of claim 1 wherein said updating the best plurality of values causes asynchronously executing a second plurality of epochs based on an updated best plurality of values.

### Claim 5 (depends on 4)

The method of claim 4 wherein at least one epoch of said plurality of epochs and at least one epoch of said second plurality of epochs concurrently execute.

### Claim 6 (depends on 1)

The method of claim 1 wherein said each costing task measures said actual cost comprises said each costing task trains a machine learning (ML) algorithm.

### Claim 7 (depends on 6)

The method of claim 6 wherein said actual cost is based on at least one of: time or memory spent training said ML algorithm, time or memory spent testing said ML algorithm, and/or accuracy indicated by said testing.

### Claim 8 (depends on 7)

The method of claim 7 further comprising estimating a predicted cost of a pending asynchronous costing task based on a plurality of actual costs that were measured.

### Claim 9 (depends on 8)

The method of claim 8 further comprising assigning a respective plurality of asynchronous costing tasks to each computer of a plurality of computers, wherein said respective plurality of asynchronous costing tasks has at least one of: a same combined predicted cost and/or a same average predicted cost.

### Claim 10 (depends on 6)

The method of claim 6 wherein said ML algorithm comprises at least one of: an artificial neural network, a random forest, and/or a support vector machine.

### Claim 11 (depends on 1)

The method of claim 1 wherein said value for said distinct hyperparameter of said exploratory plurality of values of said second task is based on a gradient of actual costs that were measured for tasks of said plurality of asynchronous costing tasks of said each epoch that already executed.

### Claim 12

One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause:
storing: a) a best cost achieved by executing a target machine learning (ML) model based on a best plurality of values respectively associated with a plurality of hyperparameters, and b) said best plurality of values;
asynchronously executing a plurality of epochs, wherein:
each epoch of the plurality of epochs is associated with a distinct hyperparameter of the plurality of hyperparameters;
each epoch of the plurality of epochs comprises a plurality of asynchronous costing tasks;
each costing task of the plurality of asynchronous costing tasks comprises an exploratory plurality of values respectively associated with the plurality of hyperparameters and based on said best plurality of values; and
the exploratory plurality of values and the best plurality of values have different values for said distinct hyperparameter;
updating the best cost and the best plurality of values when said each costing task measures an actual cost for said exploratory plurality of values that is less than the best cost.

### Claim 13 (depends on 12)

The one or more non-transitory computer-readable media of claim 12 further comprising canceling an epoch of the plurality of epochs that is selected based on at least one of: an age of the epoch, or an actual cost measured by a costing task of the plurality of asynchronous costing tasks of the epoch.

### Claim 14 (depends on 12)

The one or more non-transitory computer-readable media of claim 12 wherein:
updating the best plurality of values comprises asynchronously executing a second plurality of epochs based on an updated best plurality of values so long as at least one condition is satisfied of:
an updated best cost is less than said best cost by at least a first threshold;
a second threshold is exceeded by a difference between: a) a value of a particular hyperparameter of the updated best plurality of values and b) a value of said particular hyperparameter of the best plurality of values.

### Claim 15 (depends on 12)

The one or more non-transitory computer-readable media of claim 12 wherein said each costing task measures said actual cost comprises said each costing task trains a machine learning (ML) algorithm.

### Claim 16 (depends on 15)

The one or more non-transitory computer-readable media of claim 15 wherein said actual cost is based on at least one of: time or memory spent training said ML algorithm, time or memory spent testing said ML algorithm, and/or accuracy indicated by said testing.

### Claim 17 (depends on 16)

The one or more non-transitory computer-readable media of claim 16 further comprising estimating a predicted cost of a pending asynchronous costing task based on a plurality of actual costs that were measured.

### Claim 18 (depends on 17)

The one or more non-transitory computer-readable media of claim 17 wherein the instructions further cause assigning a respective plurality of asynchronous costing tasks to each computer of a plurality of computers, wherein said respective plurality of asynchronous costing tasks has at least one of: a same combined predicted cost and/or a same average predicted cost.

### Claim 19 (depends on 15)

The one or more non-transitory computer-readable media of claim 15 wherein said ML algorithm comprises at least one of: an artificial neural network, a random forest, and/or a support vector machine.

### Claim 20 (depends on 12)

The one or more non-transitory computer-readable media of claim 12 wherein said value for said distinct hyperparameter of said exploratory plurality of values of said second task is based on a gradient of actual costs that were measured for tasks of said plurality of asynchronous costing tasks of said each epoch that already executed.
