US 12,585,960 B2Granted

Dynamically tuning hyperparameters during ML model training

AssigneeInternational Business Machines Corporation

InventorsYuan-Chi Chang, Venkata Nagaraju Pavuluri, Dharmashankar Subramanian, Timothy Rea Dinger

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method of automatically tuning hyperparameters includes receiving a hyperparameter tuning strategy. Upon determining that one or more computing resources exceed their corresponding predetermined quota, the hyperparameter tuning strategy is rejected. Upon determining that the one or more computing resources do not exceed their corresponding predetermined quota, a machine learning model training is run with a hyperparameter point. Upon determining that one or more predetermined computing resource usage limits are exceeded for the hyperparameter point, the running of the machine learning model training is terminated for the hyperparameter point and the process returns to running the machine learning model training with a new hyperparameter point. Upon determining that training the machine learning model is complete, training results are collected and computing resource utilization metrics are determined. A correlation of the hyperparameters to the computing resource utilization is determined from the completed training of the machine learning model.

Claims19

Dependent claims

A computing device, comprising:

a processor;
a network interface coupled to the processor to enable communication over a network;
a storage device for content and programming coupled to the processor;
and
a hyperparameter engine stored in the storage device, wherein an execution of the hyperparameter engine by the processor configures the computing device to perform operations comprising:
receiving a first hyperparameter tuning strategy;
based on determining that one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, rejecting the first hyperparameter tuning strategy and receive a second hyperparameter tuning strategy different from the first hyperparameter tuning strategy,
wherein the determining, that the one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, is based on each hyperparameter point of the first hyperparameter tuning strategy being within an admissible region of a geometric convex hull in a hyperparameter space of the first hyperparameter tuning strategy;
based on determining that the one or more computing resources do not exceed their corresponding predetermined quota for the second hyperparameter tuning strategy, running a machine learning model training with a first hyperparameter point of the second hyperparameter tuning strategy;
monitoring, during each iteration of the machine learning model training, computing resource utilization for the first hyperparameter point;
based on determining that the computing resource utilization exceeds one or more predetermined computing resource usage limits for the first hyperparameter point:
terminating the running of the machine learning model training for the first hyperparameter point;
updating the admissible region based on the terminating of the machine learning model training, wherein the first hyperparameter point lies outside of the updated admissible region;
and
running the machine learning model training with a second hyperparameter point, different from the first hyperparameter point, of the second hyperparameter tuning strategy;
based on determining that the machine learning model training is complete, collecting training results and determining computing resource utilization metrics;
learning, from the updated admissible region and the completion of the machine learning model training, a correlation of one or more hyperparameters of the second hyperparameter tuning strategy to the computing resource utilization;
and
utilizing the learned correlation of the one or more hyperparameters to select hyperparameter points for subsequent machine leaning training runs.

16 more elements

A non-transitory computer readable storage medium tangibly embodying a computer readable program code having computer readable instructions that, when executed, causes a computing device to carry out a method of automatically tuning hyperparameters, the method comprising:

receiving a first hyperparameter tuning strategy;
based on determining that one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, rejecting the first hyperparameter tuning strategy and receiving a second hyperparameter tuning strategy different from the first hyperparameter tuning strategy,
wherein the determining, that the one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, is based on each hyperparameter point of the first hyperparameter tuning strategy being within an admissible region of a geometric convex hull in a hyperparameter space of the first hyperparameter tuning strategy;
based on determining that the one or more computing resources do not exceed their corresponding predetermined quota for the second hyperparameter tuning strategy, running a machine learning model training with a first hyperparameter point of the second hyperparameter tuning strategy;
monitoring, during each iteration of the machine learning model training, computing resource utilization for the first hyperparameter point;
based on determining that the computing resource utilization exceeds one or more predetermined computing resource usage limits for the first hyperparameter point:
terminating the running of the machine learning model training for the first hyperparameter point;
updating the admissible region based on the termination of the machine learning model training, wherein the first hyperparameter point lies outside of the updated admissible region;
and
running the machine learning model training with a second hyperparameter point, different from the first hyperparameter point, of the second hyperparameter tuning strategy;
based on determining that the machine learning model training is complete, collecting training results and determining computing resource utilization metrics;
learning, from the updated admissible region and the completion of the machine learning model training, a correlation of one or more hyperparameters of the second hyperparameter tuning strategy to the computing resource utilization;
and
utilizing the learned correlation of the one or more hyperparameters to select hyperparameter points for subsequent machine learning training runs.

11 more elements

A computer implemented method of automatically tuning hyperparameters, the computer implemented method comprising:

receiving a first hyperparameter tuning strategy;
based on determining that one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, rejecting the first hyperparameter tuning strategy and receiving a second hyperparameter tuning strategy different from the first hyperparameter tuning strategy,
wherein the determining, that the one or more computing resources exceed their corresponding predetermined quota for the first hyperparameter tuning strategy, is based on each hyperparameter point of the first hyperparameter tuning strategy being within an admissible region of a geometric convex hull in a hyperparameter space of the first hyperparameter tuning strategy;
based on determining that the one or more computing resources do not exceed their corresponding predetermined quota for the second hyperparameter tuning strategy, running a machine learning model training with a first hyperparameter point of the second hyperparameter tuning strategy;
monitoring, during each iteration of the machine learning model training, computing resource utilization for the first hyperparameter point;
based on determining that the computing resource utilization exceeds one or more predetermined computing resource usage limits for the first hyperparameter point:
terminating the running of the machine learning model training for the first hyperparameter point;
updating the admissible region based on the termination of the machine learning model training, wherein the first hyperparameter point lies outside of the updated admissible region;
and
running the machine learning model training with a second hyperparameter point, different from the first hyperparameter point, of the second hyperparameter tuning strategy;
based on determining that the machine learning model training is complete, collecting training results and determining computing resource utilization metrics;
learning, from the updated admissible region and the completion of the machine learning model training, a correlation of one or more hyperparameters of the second hyperparameter tuning strategy to the computing resource utilization;
and
utilizing the learned correlation of the one or more hyperparameters to select hyperparameter points for subsequent machine learning training runs.

11 more elements

Dependencies are inferred from the claim text.