Predicting neural network performance using neural network gaussian process

One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for predicting performance of a neural network, wherein the neural network has a set of network parameters, the operation comprising:

• receiving a training data set having a set of training samples, each of the training samples having a training input and a corresponding training output;
• receiving a validation data set having a set of validation pairs, each of the validation pairs having a validation input and a corresponding validation output;
• initializing (i) a validation-training kernel matrix representing similarities of the validation inputs in the validation data set and the training inputs in the training data set and (ii) a training-training kernel matrix representing similarities across the training inputs within the training data set;
• generating a final updated validation-training kernel matrix and a final updated training-training kernel matrix, the generating comprising, at each of N steps:
• sampling a set of parameter values for the set of network parameters from an initialization distribution,
• generating, from the training inputs, training feature vectors using the neural network in accordance with the sampled parameter values of the network parameters, generating, from the validation inputs, validation feature vectors using the neural network in accordance with the sampled parameter values of the network parameters, and
• updating the validation-training kernel matrix and the training-training kernel matrix using the validation feature vectors and the training feature vectors;
• performing the following operations at least once:
• generating predicted validation outputs for the validation inputs using the final updated training-training kernel matrix, the final updated validation-training kernel matrix, and the training outputs in the training data set, and
• updating an accuracy score of the neural network based on the predicted validation outputs and the validation outputs in the validation data set;
• and
• outputting the updated accuracy score as a final accuracy score that represents performance of the neural network.

A system comprising one or more computers and one or more non-transitory computer storage media encoded with instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for predicting performance of a neural network, wherein the neural network has a set of network parameters, the operation comprising:

• receiving a training data set having a set of training samples, each of the training samples having a training input and a corresponding training output;
• receiving a validation data set having a set of validation pairs, each of the validation pairs having a validation input and a corresponding validation output;
• initializing (i) a validation-training kernel matrix representing similarities of the validation inputs in the validation data set and the training inputs in the training data set and (ii) a training-training kernel matrix representing similarities across the training inputs within the training data set;
• generating a final updated validation-training kernel matrix and a final updated training-training kernel matrix, the generating comprising, at each of N steps:
• sampling a set of parameter values for the set of network parameters from an initialization distribution,
• generating, from the training inputs, training feature vectors using the neural network in accordance with the sampled parameter values of the network parameters, generating, from the validation inputs, validation feature vectors using the neural network in accordance with the sampled parameter values of the network parameters, and
• updating the validation-training kernel matrix and the training-training kernel matrix using the validation feature vectors and the training feature vectors;
• performing the following operations at least once:
• generating predicted validation outputs for the validation inputs using the final updated training-training kernel matrix, the final updated validation-training kernel matrix, and the training outputs in the training data set, and
• updating an accuracy score of the neural network based on the predicted validation outputs and the validation outputs in the validation data set;
• and
• outputting the updated accuracy score as a final accuracy score that represents performance of the neural network.