Resource allocation for tuning hyperparameters of large-scale deep learning workloads

A computer system for hyperparameter tuning, comprising:

• one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising:
• receiving hyperparameter search input data, the hyperparameter search input data comprising a number of parallel experiments and an option to mandate a workload batch size for a hyperparameter search;
• invoking a search algorithm to generate a set of hyperparameters;
• receiving, at a hyperparameter tuning resource allocator, physical resource data regarding a cluster;
• determining, via the hyperparameter tuning resource allocator, a required workload, wherein the hyperparameter tuning resource allocator uses the received number of parallel experiments to determine the required workload;
• determining, via the hyperparameter tuning resource allocator, whether the option to mandate the workload batch size was selected;
• and
• performing, via the cluster, the number of parallel experiments to search the set of hyperparameters and to tune the set of hyperparameters, wherein the hyperparameter tuning resource allocator manages the cluster, during the performing, according to the physical resource data and the required workload and by using tensor swapping.

A computer program product for hyperparameter tuning, comprising a computer-readable storage medium having program instructions embodied therewith, the program instructions being executable by a processor to cause the processor to perform a method comprising:

• receiving hyperparameter search input data, the hyperparameter search input data comprising a number of parallel experiments and an option to mandate a workload batch size for a hyperparameter search;
• invoking a search algorithm to generate a set of hyperparameters;
• receiving, at a hyperparameter tuning resource allocator, physical resource data regarding a cluster;
• determining, via the hyperparameter tuning resource allocator, a required workload, wherein the hyperparameter tuning resource allocator uses the received number of parallel experiments to determine the required workload;
• determining, via the hyperparameter tuning resource allocator, whether the option to mandate the workload batch size was selected;
• and
• performing, via the cluster, the number of parallel experiments to search the set of hyperparameters and to tune the set of hyperparameters, wherein the hyperparameter tuning resource allocator manages the cluster, during the performing, according to the physical resource data and the required workload and by using tensor swapping.