Efficient configuration selection for automated machine learning

A method comprising:

• iteratively pruning a set of machine-learning configurations based on a training dataset and a test dataset by using one or more hardware processors to perform operations comprising, in each of a plurality of iterations:
• sampling the training and test datasets in accordance with a sampling schedule associated with a machine-learning configuration selected from the set of machine-learning configurations;
• training the selected machine-learning configuration based on the sampled training dataset and determining a training accuracy associated with the trained selected machine-learning configuration;
• evaluating the trained selected machine-learning configuration based on the sampled test dataset to determine a test accuracy associated with the trained selected machine-learning configuration;
• determining a confidence interval associated with the trained selected machine-learning configuration based at least in part on the training accuracy and the test accuracy, the confidence interval providing estimated upper and lower bounds of an accuracy, the upper bound being computed from the training accuracy but not the test accuracy, and the lower bound being computed from the test accuracy but not the training accuracy;
• pruning the set of machine-learning configurations based on comparisons between the determined confidence interval and confidence intervals associated with other machine-learning configurations within the set of machine-learning configurations;
• and
• selecting one of the machine-learning configurations remaining within the pruned set for a next iteration.

A system comprising:

• one or more hardware processors configured to implement a plurality of processing components for determining an approximate best machine-learning configuration among a set of machine-learning configurations, the processing components comprising:
• a training and test component configured to, upon selection of one of the machine-learning configurations within the set of machine-learning configurations and determination of a sample size associated with the machine-learning configuration:
• cause a training dataset to be sampled in accordance with the sample size associated with the machine-learning configuration to obtain a sampled training dataset,
• train the selected machine learning configuration on the sampled training dataset to optimize a training value of a quality metric, and
• cause the trained machine-learning configuration to be tested on at least a sample of a test dataset to determine a test value of the quality metric;
• and
• a sampling and scheduling component configured to:
• compute, from the training value of the quality metric and the test value of the quality metric, a confidence interval providing upper and lower bounds for the quality metric, the upper bound being computed from the training value but not the test value, and the lower bound being computed from the test value but not the training value, and
• iteratively select machine-learning configurations for training by the training and test component and determine associated sample sizes for the selected machine-learning configurations, and prune the set of machine-learning configurations based on confidence intervals computed from training and test values of the quality metric determined by the training and test component.