Resource constrained neural network architecture search

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• defining a computational cell of a neural network, wherein the computational cell comprises a directed graph of nodes and edges, each node representing a respective neural network latent representation and each edge representing a respective operation that transforms a respective neural network latent representation;
• replacing each operation that transforms a respective neural network latent representation with a respective linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in a respective linear combination has a respective mixing weight that is parameterized by one or more computational cell hyper parameters;
• iteratively adjusting values of i) the computational cell hyper parameters, and ii) computational cell weights, to optimize a validation loss function subject to one or more computational resource constraints, comprising, for each iteration:
• performing an unconstrained optimization of the validation loss function to update values of the computational cell hyper parameters for a previous iteration and to obtain adjusted values of the computational cell weights;
• and
• projecting the updated values of the computational cell hyper parameters to a nearest point in a feasible set defined by the one or more resource constraints to obtain adjusted values of the computational cell hyper parameters;
• and
• generating a neural network for performing a machine learning task using the defined computational cell and the adjusted values of the computational cell hyper parameters and computational cell weights.

A computer-readable storage medium comprising instructions stored thereon that are executable by a processing device and upon such execution cause the processing device to perform operations comprising:

• defining a computational cell of a neural network, wherein the computational cell comprises a directed graph of nodes and edges, each node representing a respective neural network latent representation and each edge representing a respective operation that transforms a respective neural network latent representation;
• replacing each operation that transforms a respective neural network latent representation with a respective linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in a respective linear combination has a respective mixing weight that is parameterized by one or more computational cell hyper parameters;
• iteratively adjusting values of i) the computational cell hyper parameters, and ii) computational cell weights, to optimize a validation loss function subject to one or more computational resource constraints, comprising, for each iteration:
• performing an unconstrained optimization of the validation loss function to update values of the computational cell hyper parameters for a previous iteration and to obtain adjusted values of the computational cell weights;
• and
• projecting the updated values of the computational cell hyper parameters to a nearest point in a feasible set defined by the one or more resource constraints to obtain adjusted values of the computational cell hyper parameters;
• and
• generating a neural network for performing a machine learning task using the defined computational cell and the adjusted values of the computational cell hyper parameters and computational cell weights.