Using hierarchical representations for neural network architecture searching

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for automatically determining a neural network architecture, the operations comprising:

• generating data representing the neural network architecture as a data structure defining a hierarchical set of graphs comprising a succession of hierarchical levels of graphs, each graph having an input, an output, a plurality of nodes between the input and the output, and one or more edges that each connect two respective nodes, wherein:
• a node corresponds to a feature map within a neural network defined by the neural network architecture, each edge connects an input node of the edge to an output node of the edge and corresponds to an operation performed on a feature map of the input node of the edge to provide a feature map to the output node of the edge, such that each graph performs an operation, the succession of hierarchical levels include a lowest hierarchical level and a plurality of additional hierarchical levels, generating the data comprises defining the hierarchical set of graphs, and defining the hierarchical set of graphs comprises:
• defining the lowest hierarchical level comprising one or more lowest level graphs, wherein operations corresponding to edges of the one or more lowest level graphs are selected from a set of primitive neural network operations, and
• recursively defining the additional hierarchical levels, wherein each respective additional hierarchical level comprises a respective set of one or more graphs, wherein recursively defining each respective additional hierarchical level comprises, for each graph in the one or more graphs in the respective additional hierarchical level, selecting a set of lower-level graphs from the graphs in the hierarchical level that precedes the respective additional hierarchical level in the succession of hierarchical levels, and generating data representing the graph by assembling the lower-level graphs selected from the graphs in the hierarchical level that precedes the respective additional hierarchical level;
• initiating a population of two or more sample neural network architectures based on the data structure defining the hierarchical set of graphs, wherein each respective sample neural network architecture in the population is initiated by modifying one or more of the operations performed by the edges of the hierarchical set of graphs;
• generating sample neural networks having the sample neural network architectures in the population;
• training the sample neural networks;
• evaluating the sample neural networks by determining a fitness value for each of the sample neural networks;
• and
• selecting one or more of the sample neural network architectures in the population according to the determined fitness values to determine a neural network architecture.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for automatically determining a neural network architecture, the operations comprising:

• generating data representing the neural network architecture as a data structure defining a hierarchical set of graphs comprising a succession of hierarchical levels of graphs, each graph having an input, an output, a plurality of nodes between the input and the output, and one or more edges that each connect two respective nodes, wherein:
• a node corresponds to a feature map within a neural network defined by the neural network architecture, each edge connects an input node of the edge to an output node of the edge and corresponds to an operation performed on a feature map of the input node of the edge to provide a feature map to the output node of the edge, such that each graph performs an operation, the succession of hierarchical levels include a lowest hierarchical level and a plurality of additional hierarchical levels, generating the data comprises defining the hierarchical set of graphs, and defining the hierarchical set of graphs comprises:
• defining the lowest hierarchical level comprising one or more lowest level graphs, wherein operations corresponding to edges of the one or more lowest level graphs are selected from a set of primitive neural network operations, and
• recursively defining the additional hierarchical levels, wherein each respective additional hierarchical level comprises a respective set of one or more graphs, wherein recursively defining each respective additional hierarchical level comprises, for each graph in the one or more graphs in the respective additional hierarchical level, selecting a set of lower-level graphs from the graphs in the hierarchical level that precedes the respective additional hierarchical level in the succession of hierarchical levels, and generating data representing the graph by assembling the lower-level graphs selected from the graphs in the hierarchical level that precedes the respective additional hierarchical level;
• initiating a population of two or more sample neural network architectures based on the data structure defining the hierarchical set of graphs, wherein each respective sample neural network architecture in the population is initiated by modifying one or more of the operations performed by the edges of the hierarchical set of graphs;
• generating sample neural networks having the sample neural network architectures in the population;
• training the sample neural networks;
• evaluating the sample neural networks by determining a fitness value for each of the sample neural networks;
• and
• selecting one or more of the sample neural network architectures in the population according to the determined fitness values to determine a neural network architecture.