Determining hyperparameters using sequence generation neural networks

A method performed by one or more computers, the method comprising:

• receiving metadata specifying (i) a machine learning task and (ii) a set of hyperparameters for training a machine learning model to perform the machine learning task;
• generating a metadata sequence that represents the metadata as a sequence of tokens from a vocabulary of tokens;
• and
• determining a respective optimized value for each of the hyperparameters in the set, comprising:
• at each of a plurality of iterations:
• generating one or more trials that each specify a respective value for each of the hyperparameters in the set, comprising, for each trial:
• generating an input sequence for the iteration that comprises (i) the metadata sequence and (ii) for any earlier trials performed at any preceding iterations, a respective sequence that represents the respective values for the hyperparameters specified by the earlier trial and a measure of performance for the trial as a sequence of tokens from the vocabulary of tokens,
• determining a plurality of candidate trials that each specify a different combination of respective values for the hyperparameters in the set;
• for each of the candidate trials, processing an input sequence for the candidate trial that comprises (i) the input sequence for the iteration and (ii) a sequence that represents the respective values for the hyperparameters that are specified by the candidate trial using a sequence generation neural network to generate an output that represents a predicted performance measure for the candidate trial;
• and
• selecting, as the hyperparameter values specified by the trial, the hyperparameter values specified by one of the candidate trials based on the predicted performance measures;
• for each trial, training the machine learning model on a set of training data for the machine learning task with the set of hyperparameters having the respective values specified by the trial and determining a performance measure for the trial based on a performance of the trained machine learning model;
• and
• selecting, as the respective optimized values for the hyperparameters, the respective values specified by one of the trials generated at one of the plurality of iterations based on the performance measures for the trials.

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• receiving metadata specifying (i) a machine learning task and (ii) a set of hyperparameters for training a machine learning model to perform the machine learning task;
• generating a metadata sequence that represents the metadata as a sequence of tokens from a vocabulary of tokens;
• and
• determining a respective optimized value for each of the hyperparameters in the set, comprising:
• at each of a plurality of iterations:
• generating one or more trials that each specify a respective value for each of the hyperparameters in the set, comprising, for each trial:
• generating an input sequence for the iteration that comprises (i) the metadata sequence and (ii) for any earlier trials performed at any preceding iterations, a respective sequence that represents the respective values for the hyperparameters specified by the earlier trial and a measure of performance for the trial as a sequence of tokens from the vocabulary of tokens, and
• processing an input sequence for the trial that comprises the input sequence for the iteration using a sequence generation neural network to generate an output sequence for the trial that represents respective values for the hyperparameters in the set as tokens from the vocabulary of tokens;
• and
• for each trial, training the machine learning model on a set of training data for the machine learning task with the set of hyperparameters having the respective values specified by the trial and determining a performance measure for the trial based on a performance of the trained machine learning model;
• and
• selecting, as the respective optimized values for the hyperparameters, the respective values specified by one of the trials generated at one of the plurality of iterations based on the performance measures for the trials.