Neural Architecture Search with Improved Computational Efficiency
AssigneeGoogle Inc.
InventorsDa Huang, Chengrun Yang, Pieter-Jan Kindermans, Hanxiao Liu, Quoc V. Le, Madeleine Richards Udell, Yifeng Lu, Gabriel Mintzer Bender
Life of this patent
Estimating the term…
Abstract
Provided are neural architecture search techniques that have improved computational efficiency via performance of an initial constraint evaluation and improved gradient update approach. Further, the proposed approaches provide significant improvements for certain modalities of input data, such as tabular datasets.
Claims20
A computer-implemented method of neural architecture search with increased computational efficiency, the method comprising:
- defining, by a computing system comprising one or more computing devices, a plurality of searchable parameters that control an architecture of a neural network, wherein the neural network is configured to process input data to produce inferences;
- and
- for one or more iterations:
- determining, by the computing system using a controller model, a new set of values for the plurality of searchable parameters to generate a new architecture for the neural network;
- determining, by the computing system, whether the neural network with the new architecture satisfies one or more constraints;
- when the neural network with the new architecture does not satisfy the one or more constraints:
- discarding, by the computing system, the new architecture;
- and
- when the neural network with the new architecture satisfies the one or more constraints:
- determining, by the computing system, one or more performance metrics for the neural network with the new architecture relative to production of inferences for a set of validation data;
- evaluating, by the computing system, a value function that provides a value based at least in part on the one or more performance metrics and a conditional probability of the new architecture for the neural network given that the new architecture for the neural network satisfies the one or more constraints;
- and
- updating, by the computing system, one or more values of one or more parameters of the controller model based on the value function.
A computer system, comprising:
- one or more processors;
- and
- one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
- determining, by the computing system using a controller model, a new set of values for a plurality of searchable parameters to generate a new architecture for the neural network;
- determining, by the computing system, one or more performance metrics for the neural network with the new architecture relative to production of inferences for a set of validation data;
- evaluating, by the computing system, a value function that provides a value based at least in part on the one or more performance metrics and a conditional probability of the new architecture for the neural network given that the new architecture for the neural network satisfies one or more constraints;
- and
- updating, by the computing system, one or more values of one or more parameters of the controller model based on the value function.
One or more non-transitory computer-readable media that collectively store a neural network having a final architecture identified by performance of operations for a plurality of iterations, the operations comprising:
- determining, by the computing system using a controller model, a new set of values for a plurality of searchable parameters to generate a new architecture for the neural network;
- determining, by the computing system, one or more performance metrics for the neural network with the new architecture relative to production of inferences for a set of validation data;
- evaluating, by the computing system, a value function that provides a value based at least in part on the one or more performance metrics and a conditional probability of the new architecture for the neural network given that the new architecture for the neural network satisfies one or more constraints;
- and
- updating, by the computing system, one or more values of one or more parameters of the controller model based on the value function.
Dependencies are inferred from the claim text.