Multi-objective distributed hyperparameter tuning system

A computing device comprising:

• a processor;
• and
• a computer-readable medium operably coupled to the processor, the computer-readable medium having computer-readable instructions stored thereon that, when executed by the processor, cause the computing device to
• receive a selection of a plurality of search methods;
• (A) determine a plurality of hyperparameter configurations using a search method of a search method type selected from the plurality of search methods, wherein each hyperparameter configuration of the plurality of hyperparameter configurations includes a value for each hyperparameter of a plurality of hyperparameters, wherein each hyperparameter configuration of the plurality of hyperparameter configurations is a unique combination of values for the plurality of hyperparameters, wherein the plurality of hyperparameter configurations are identified based on a predefined model type;
• (B) select a hyperparameter configuration from the plurality of hyperparameter configurations;
• (C) determine if the selected hyperparameter configuration satisfies a linear constraint on at least one hyperparameter of the plurality of hyperparameters;
• (D) when the selected hyperparameter configuration does not satisfy the linear constraint,
• determine if a projection of the selected hyperparameter configuration is included in a first cache, wherein the first cache stores pairs of hyperparameter configurations, wherein each pair of the pairs includes a previously selected hyperparameter configuration and a projected hyperparameter configuration computed from the previously selected hyperparameter configuration, wherein determining if the projection of the selected hyperparameter configuration is included in the first cache includes a comparison of the selected hyperparameter configuration with the previously selected hyperparameter configuration of each pair stored in the first cache;
• when the projection is included in the first cache, extract the projected hyperparameter configuration of a matching pair of the pairs from the first cache using the selected hyperparameter configuration and replace the selected hyperparameter configuration with the extracted projected hyperparameter configuration of the matching pair of the pairs in the plurality of hyperparameter configurations;
• when the projection is not included in the first cache,
• assign a projection computation for the selected hyperparameter configuration to a session;
• receive a computed projected hyperparameter configuration from the session for the selected hyperparameter configuration when the projection computation is complete by the session;
• store the received computed projected hyperparameter configuration and the selected hyperparameter configuration to the first cache;
• and
• replace the selected hyperparameter configuration with the received computed projected hyperparameter configuration in the plurality of hyperparameter configurations;
• (E) repeat (B) through (D) until each hyperparameter configuration of the plurality of hyperparameter configurations in (A) is selected;
• (F) compute an objective function value for each hyperparameter configuration of the plurality of hyperparameter configurations using a predefined objective function by training a model of the predefined model type using a respective hyperparameter configuration of the plurality of hyperparameter configurations and scoring the trained model;
• (G) repeat (A) through (F) until each search method of the plurality of search methods is selected in (A);
• (H) identify a best hyperparameter configuration of the plurality of hyperparameter configurations based on an extremum value of the computed objective function value for each hyperparameter configuration of the plurality of hyperparameter configurations;
• and
• (I) output the identified best hyperparameter configuration.

A method of selecting tuned hyperparameter values for training a machine learning model, the method comprising:

• receiving, by a computing device, a selection of a plurality of search methods;
• (A) determining, by the computing device, a plurality of hyperparameter configurations using a search method of a search method type selected from the plurality of search methods, wherein each hyperparameter configuration of the plurality of hyperparameter configurations includes a value for each hyperparameter of a plurality of hyperparameters, wherein each hyperparameter configuration of the plurality of hyperparameter configurations is a unique combination of values for the plurality of hyperparameters, wherein the plurality of hyperparameter configurations are identified based on a predefined model type;
• (B) selecting, by the computing device, a hyperparameter configuration from the plurality of hyperparameter configurations;
• (C) determining, by the computing device, if the selected hyperparameter configuration satisfies a linear constraint on at least one hyperparameter of the plurality of hyperparameters;
• (D) when the selected hyperparameter configuration does not satisfy the linear constraint,
• determining, by the computing device, if a projection of the selected hyperparameter configuration is included in a first cache, wherein the first cache stores pairs of hyperparameter configurations, wherein each pair of the pairs includes a previously selected hyperparameter configuration and a projected hyperparameter configuration computed from the previously selected hyperparameter configuration, wherein determining if the projection of the selected hyperparameter configuration is included in the first cache includes a comparison of the selected hyperparameter configuration with the previously selected hyperparameter configuration of each pair stored in the first cache;
• when the projection is included in the first cache, extracting, by the computing device, the projected hyperparameter configuration of a matching pair of the pairs from the first cache using the selected hyperparameter configuration and replace the selected hyperparameter configuration with the extracted projected hyperparameter configuration of the matching pair of the pairs in the plurality of hyperparameter configurations;
• when the projection is not included in the first cache,
• assigning, by the computing device, a projection computation for the selected hyperparameter configuration to a session;
• receiving, by the computing device, a computed, projected hyperparameter configuration from the session for the selected hyperparameter configuration when the projection computation is complete by the session;
• storing, by the computing device, the received, computed, projected hyperparameter configuration and the selected hyperparameter configuration to the first cache;
• and
• replacing, by the computing device, the selected hyperparameter configuration with the received, computed, projected hyperparameter configuration in the plurality of hyperparameter configurations;
• (E) repeating, by the computing device, (B) through (D) until each hyperparameter configuration of the plurality of hyperparameter configurations in (A) is selected;
• (F) computing, by the computing device, an objective function value for each hyperparameter configuration of the plurality of hyperparameter configurations using a predefined objective function by training a model of the predefined model type using a respective hyperparameter configuration of the plurality of hyperparameter configurations and scoring the trained model;
• (G) repeating, by the computing device, (A) through (F) until each search method of the plurality of search methods is selected in (A);
• (H) identifying, by the computing device, a best hyperparameter configuration of the plurality of hyperparameter configurations based on an extremum value of the computed objective function value for each hyperparameter configuration of the plurality of hyperparameter configurations;
• and
• (I) outputting, by the computing device, the identified best hyperparameter configuration.