Multi-modal neural network architecture search

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• receiving training data for training a neural network to perform a multi-modal machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples, wherein each training example comprises a respective network input from each of a plurality of modalities;
• and
• determining, using the training data, an optimized neural network architecture for performing the machine learning task, comprising:
• generating a plurality of candidate neural network architectures, wherein:
• each candidate neural network architecture comprises a plurality of operation blocks that each receive a respective block input comprising at least a respective first input hidden state and a respective second hidden state and apply one or more operations to the block input to generate an output hidden state, and
• generating each of the plurality of candidate neural network architectures comprises, for each of the operation blocks in the candidate neural network architecture:
• selecting, for each of the plurality of operation blocks, each of the first and second hidden states that are received as input by the operation block from a respective set of possible inputs that includes at least each of the respective network inputs from each of the plurality of modalities;
• determining, for each candidate network architecture, a respective fitness on the multi-modal machine learning task;
• and
• selecting, from the candidate neural network architectures, the optimized neural network architecture based on the respective fitness.

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• receiving training data for training a neural network to perform a multi-modal machine learning task, the training data comprising a plurality of training examples and a respective target output for each of the training examples, wherein each training example comprises a respective network input from each of a plurality of modalities;
• and
• determining, using the training data, an optimized neural network architecture for performing the machine learning task, comprising:
• generating a plurality of candidate neural network architectures, wherein:
• each candidate neural network architecture comprises a plurality of operation blocks that each receive a respective block input comprising at least a respective first input hidden state and a respective second hidden state and apply one or more operations to the block input to generate an output hidden state, and
• generating each of the plurality of candidate neural network architectures comprises, for each of the operation blocks in the candidate neural network architecture:
• selecting, for each of the plurality of operation blocks, each of the first and second hidden states that are received as input by the operation block from a respective set of possible inputs that includes at least each of the respective network inputs from each of the plurality of modalities;
• determining, for each candidate network architecture, a respective fitness on the multi-modal machine learning task;
• and
• selecting, from the candidate neural network architectures, the optimized neural network architecture based on the respective fitness